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Abstract 


This  study  evaluated  the  utility  and  ease  of  use  of  features  of  the  Tactical  Battlefield  Command 
Systems  (TBCS)/Chameleon  using  participants  representing  command  elements  of  a  combat  team. 
Seven  participants  role-played  an  advance  to  contact  scenario  developed  by  Joint  Command  Staff 
Training  Centre  (JCSTC)  in  13  segments.  Following  each  segment,  participants  provided  user 
feedback  on  25  key  features  and  tools  of  the  software. 

The  overall  results  indicated  that  the  features  and  tools  in  TBCS/Chameleon  are  seen  to  be  generally 
useful  by  the  combat  team  across  a  range  of  activities.  Many  specific  features  currently  in  the 
software,  as  well  as  future  features,  were  seen  to  have  particularly  high  utility  and  have  the  potential  to 
improve  situation  awareness,  reduce  workload,  improve  communication  effectiveness  and  support 
decision-making. 

Flowever,  there  are  a  number  of  areas  in  which  the  utility  of  features  can  be  improved.  Specific 
recommendations  are  made  to  support  these  improvements  across  a  range  of  features  including:  map 
use,  communication  tools,  production  of  orders  and  access  to  information. 

These  recommendations  concentrate  on  utility  issues  with  a  secondary  focus  on  increasing  the  ease  of 
use  of  some  features. 

The  user  review  process  should  continue  at  each  major  build  of  the  TBCS/Chameleon.  As  the 
development  moves  from  a  concept  based  development  to  a  fieldable  system  the  user  reviews  should 
move  from  utility  based  to  usability  based.  Tabletop  user  reviews  of  concepts  will  also  assist  with 
design  decisions  between  major  builds. 
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Resume 


Cette  etude  a  evalue  l’utilite  et  la  convivialite  des  fonctions  du  Systeme  tactique  de  commandement 
sur  le  champ  de  bataille  (STCCB)/Chameleon  en  faisant  appel  a  des  participants  representant  les 
elements  de  commandement  d’une  equipe  de  combat.  Sept  participants  ont  simule  un  scenario  de 
marche  a  l’ennemi  mise  au  point  par  le  Centre  de  formation  de  commandement  et  d’etat-major 
interarmees  (CFCEMI)  en  13  segments.  Apres  chaque  segment,  les  participants  ont  foumi  une 
retroaction  sur  25  caracteristiques  et  outils  principaux  du  logiciel. 

Les  resultats  globaux  ont  indique  que  les  caracteristiques  et  outils  du  STCCB/Chameleon  sont 
consideres  comme  generalement  utiles  par  l’equipe  de  combat,  et  ce,  dans  differents  champs 
d’activite.  De  nombreuses  caracteristiques  actuelles  du  logiciel,  de  meme  que  des  caracteristiques 
envisagees,  sont  per£ues  comme  etant  particulierement  utiles  et  comme  ayant  le  potentiel  d’ameliorer 
la  connaissance  de  la  situation,  de  reduire  la  charge  de  travail,  d’accroitre  Tefficacite  des 
communications  et  de  soutenir  la  prise  de  decisions. 

Cependant,  il  y  a  un  certain  nombre  des  caracteristiques  qui  pourraient  etre  ameliorees.  Des 
recommandations  specifiques  sont  formulees  en  vue  d’operer  ces  ameliorations  a  diverses 
caracteristiques  :  utilisation  de  cartes,  outils  de  communication,  production  d’ordres  et  acces  a 
1’ information. 

Ces  recommandations  sont  axees  sur  des  questions  utilitaires  et  ont  pour  deuxieme  centre  d’interet  la 
convivialite  accrue  de  certaines  fonctions. 

Le  processus  d’examen  par  les  utilisateurs  devrait  se  poursuivre  avec  chacune  des  nouvelles  versions 
importantes  du  STCCB/Chameleon.  A  mesure  que  le  systeme  passe  de  l’etape  conceptuelle  a  l’etape 
de  l’utilisation  sur  le  terrain,  les  examens  par  les  utilisateurs  devraient  etre  de  moins  en  moins  axes  sur 
l’utilite  du  systeme  et  de  plus  en  plus  sur  sa  convivialite.  L’examen  de  concepts  par  les  utilisateurs  au 
moyen  de  simulations  facilitera  egalement  la  prise  de  decisions  conceptuelles  entre  les  principales 
versions. 
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Executive  Summary 


This  report  details  the  purpose,  method,  results  and  conclusions  of  the  Tactical  Battlefield  Command 
Systems  (TBCS)/Chameleon  Utility  Trial.  Participants  representing  a  combat  team  evaluated  the 
utility  and  ease  of  use  of  features  present  in  Version  3. 1  of  the  software. 

The  purpose  of  this  trial  was  to  provide  systematic,  useful  and  timely  user  feedback  to  the  design 
team,  in  order  to  enhance  the  utility  and  functionality  of  the  TBCS/Chameleon  system  and  to  assist 
with  the  direction  of  future  system  developments.  The  specific  objectives  of  the  trial  were  to: 

•  Provide  user  feedback  on  the  utility  of  the  features  in  TBCS/Chameleon  as  they  apply  to 
various  combat  team  levels  of  command,  and 

•  Provide  user  feedback  on  the  ease  of  use  of  the  system’s  interface  and  functionality. 

Seven  participants  representing  command  elements  of  a  cross  section  of  a  combat  team  were  trained 
for  five  hours  on  how  to  use  the  basic  features  of  TBCS/Chameleon  software.  They  then  role-played 
an  advance  to  contact  scenario  developed  by  Joint  Command  Staff  Training  Centre  (JCSTC)  in  13 
segments.  Rating  data  for  the  key  features  and  tools  of  the  software  were  provided  by  participants 
following  each  segment.  Other  data  were  obtained  through  focus  groups  following  each  stage,  as  well 
as  through  observations  by  the  trial  administrators.  A  final  focus  group  discussion  was  held  at  the 
conclusion  of  the  3  day  trial  to  review  overall  impressions  of  the  system  functionality  and  to  discuss 
problem  areas. 

The  overall  results  indicated  that  the  features  and  tools  in  TBCS/Chameleon  are  seen  to  be  generally 
useful  by  the  combat  team  across  a  range  of  activities.  Many  specific  features  currently  in  the 
software,  as  well  as  future  features,  were  seen  to  have  particularly  high  utility  and  have  the  potential  to 
improve  situation  awareness,  reduce  workload,  improve  communication  effectiveness  and  support 
decision-making.  Some  of  the  most  positive  aspects  of  the  system  were: 

•  The  enhancement  to  situation  awareness  provided  by  the  ability  to  plot  unit  locations,  contacts 
and  other  similar  information  (including  GPS  data)  directly  on  the  map. 

•  Improved  effectiveness  and  accuracy  with  which  orders  can  be  received,  prepared  and 
transmitted. 

•  The  streamlining  of  tasks  associated  with  the  collation  and  integration  of  information 
concerning  resources. 

•  The  enhanced  maintenance  of  situation  awareness  and  communication  effectiveness  if  the  user 
is  required  to  change  command  and  control  systems  for  example  switching  vehicles. 

•  Enhanced  situation  awareness  in  planning  operations. 

•  The  potential  for  reducing  voice  communication  and  allowing  audio  channels  to  be  reserved 
for  the  most  critical  information. 

•  The  potential  for  reducing  a  number  of  error  sources  in  the  communication  process. 

•  The  potential  to  enhance  situation  awareness  by  providing  access  to  unit  level  information 
concerning  capability/status. 

•  Considerable  potential  for  a  variety  of  TBCS/Chameleon  based  decision  aids  to  support 
tactical  planning  tasks. 

However,  there  are  a  number  of  issues  in  which  the  utility  of  features  can  be  improved.  Thirty  three 
specific  recommendations  are  made  to  support  these  improvements  across  a  range  of  features 
including:  map  use,  communication  tools,  production  of  orders  and  access  to  information. 
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Major  issues  revealed  during  the  trial  were  the  need  to: 

•  Develop  appropriate  features  and  associated  interfaces  to  address  the  specific  needs  of 
different  combat  team  members. 

•  Develop  appropriate  features  and  associated  interfaces  to  address  the  requirements  of  different 
operational  contexts. 

•  Increase  the  speed  of  task  execution  across  all  features. 

•  Improve  the  system  interface  to  enhance  situation  awareness  of  new  information. 

•  Provide  support  for  the  improved  integration  of  global  and  local  situation  awareness. 

•  Address  potentially  serious  areas  where  the  system  may  impair  situation  awareness. 

•  Enhance  the  ability  of  the  user  to  integrate  text-based  communication  with  map  information. 

The  general  impression  obtained  from  the  trial  results  is  that  the  current  feature  set  does  not  match  the 
specific  features  needed  to  support  the  most  critical  and  frequent  tasks  done  by  different  combat  team 
positions. 

The  major  limitations  in  generalising  the  trial  data  resulted  from  two  sources:  (i)  the  low  level  of 
applicable  combat  experience  in  the  trial  participants  for  the  roles  required  to  be  played,  and  (ii)  the 
lack  of  stability  of  the  system  software  which  may  have  resulted  in  user  frustration  and  response  bias. 
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Sommaire 


Ce  rapport  decrit  le  but,  la  methode,  les  resultats  et  les  conclusions  de  l’essai  d’utilite  du  Systeme 
tactique  de  commandement  sur  le  champ  de  bataille  (STCCB)/Chameleon.  Des  participants 
representant  une  equipe  de  combat  ont  evalue  Tutilite  et  la  convivialite  des  caracteristiques  de  la 
version  3.1  du  logiciel. 

Le  but  de  cet  essai  etait  de  foumir  a  1’ equipe  de  conception  une  retroaction  systematique,  utile  et 
opportune  de  la  part  des  utilisateurs  afm  d’ameliorer  l’utilite  et  la  fonctionnalite  du 
STCCB/Chameleon  et  d’orienter  les  futurs  developpements  de  systeme.  Les  objectifs  precis  de  l’essai 
etaient : 

•  de  foumir  une  retroaction  de  la  part  des  utilisateurs  sur  l’utilite  des  caracteristiques  du 
STCCB/Chameleon  pour  les  differents  niveaux  de  commandement  d’une  equipe  de  combat; 

•  de  foumir  une  retroaction  de  la  part  des  utilisateurs  sur  la  convivialite  de  Tinterface  et  des 
fonctions  du  systeme. 

Sept  participants  representant  les  elements  de  commandement  d’un  groupe  representatif  d’une  equipe 
de  combat  ont  ete  entraines  pendant  cinq  heures  a  utiliser  les  fonctionnalites  de  base  du  logiciel  du 
STCCB/Chameleon.  Ils  ont  ensuite  simule  un  scenario  de  marche  a  l’ennemi  mise  au  point  par  le 
Centre  de  formation  de  commandement  et  d’etat-major  interarmees  (CFCEMI)  en  13  segments.  Des 
scores  pour  les  principaux  outils  et  caracteristiques  du  logiciel  ont  ete  foumis  par  les  participants  apres 
chaque  segment.  D’autres  donnees  ont  ete  recueillies  au  moyen  de  groupes  de  discussion  apres  chacun 
des  stades,  ainsi  qu’au  moyen  des  observations  des  administrateurs  de  Tessai.  Une  demiere  seance  de 
discussion  a  ete  organisee  au  terme  de  Tessai  de  trois  jours  pour  passer  en  revue  les  impressions 
generates  quant  a  la  fonctionnalite  du  systeme  et  pour  discuter  des  questions  qui  posent  probleme. 

Les  resultats  globaux  ont  indique  que  les  caracteristiques  et  outils  du  STCCB/Chameleon  sont 
consideres  comme  generalement  utiles  par  Tequipe  de  combat,  et  ce,  dans  differents  champs 
d’activite.  De  nombreuses  caracteristiques  actuelles  du  logiciel,  de  meme  que  des  caracteristiques 
envisagees,  sont  perfues  comme  etant  particulierement  utiles  et  comme  ayant  le  potentiel  d’ameliorer 
la  connaissance  de  la  situation,  de  reduire  la  charge  de  travail,  d’accroitre  Tefficacite  des 
communications  et  de  soutenir  la  prise  de  decisions.  Parmi  les  aspects  les  plus  positifs  du  systeme, 
mentionnons  les  suivants  : 

•  L’ amelioration  de  la  connaissance  de  la  situation  assuree  par  la  capacite  de  pointer 

T emplacement  des  unites,  les  contacts  et  d’autres  renseignements  analogues  (y  compris  des 
donnees  GPS)  directement  sur  une  carte. 

•  L’ amelioration  de  Tefficacite  et  de  Texactitude  avec  lesquelles  des  ordres  peuvent  etre  rcgucs, 
prepares  et  transmis. 

•  La  rationalisation  des  taches  associees  au  regroupement  et  a  T integration  des  renseignements 
concemant  les  ressources. 

•  La  facilitation  du  maintien  d’une  connaissance  de  la  situation  et  d’une  communication  efficace 
si  Tutilisateur  doit  changer  de  systeme  de  commandement  et  de  controle,  en  changeant  de 
vehicule  par  exemple. 

•  Une  connaissance  de  la  situation  accrue  lors  de  la  planification  des  operations. 

•  Le  potentiel  de  reduire  le  recours  a  la  communication  en  phonie  et  de  permettre  aux  canaux 
audio  d’etre  reserves  a  la  communication  des  renseignements  essentiels. 
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•  Le  potentiel  de  reduire  un  certain  nombre  de  sources  d’erreurs  dans  le  processus  de 
communication. 

•  Le  potentiel  d’ameliorer  la  connaissance  de  la  situation  en  donnant  acces  a  de  l’information 
des  unites  concemant  les  capacites  et  le  statut. 

•  Un  potentiel  considerable  pour  que  divers  outils  d’aide  a  la  decision  du  STCCB/Chameleon 
appuient  les  taches  de  planification  tactique. 

Cependant,  il  y  a  un  certain  nombre  de  caracteristiques  qui  pourraient  etre  ameliorees.  Trente-trois 
recommandations  specifiques  sont  formulees  en  vue  d’operer  ces  ameliorations  a  diverses 
caracteristiques  :  utilisation  de  cartes,  outils  de  communication,  production  d’ordres  et  acces  a 
1’  information. 

Parmi  les  principals  ameliorations  proposees,  mentionnons  la  necessite  : 

•  de  mettre  au  point  des  caracteristiques  et  des  interfaces  adaptees  pour  repondre  aux  besoins 
particuliers  des  differents  membres  de  l’equipe  de  combat. 

•  de  mettre  au  point  des  caracteristiques  et  des  interfaces  adaptees  pour  repondre  aux  besoins  de 
differents  contextes  operationnels. 

•  d’accroitre  la  vitesse  d’ execution  des  taches  pour  1’ ensemble  des  caracteristiques. 

•  d’ameliorer  1’ interface  systeme  de  maniere  a  accroitre  la  connaissance  de  la  situation 
relativement  aux  nouveaux  renseignements. 

•  de  fournir  du  soutien  en  vue  de  l’integration  accrue  de  la  connaissance  de  la  situation  globale 
et  locale. 

•  de  corriger  les  problemes  potentiellement  serieux  qui  pourraient  nuire  a  la  connaissance  de  la 
situation. 

•  d’accroitre  la  capacite  de  l’utilisateur  d’integrer  la  communication  textuelle  et  les  donnees 
cartographiques. 

L’ impression  generale  obtenue  des  resultats  de  l’essai  est  que  la  serie  actuelle  de  caracteristiques  ne 
coincide  pas  avec  les  caracteristiques  specifiques  requises  pour  soutenir  les  taches  critiques  et 
frequentes  executees  par  les  differents  membres  de  l’equipe  de  combat. 

Les  principales  limitations  a  la  generalisation  des  resultats  de  l’essai  sont  dues  a  deux  facteurs  :  (i)  le 
peu  d’experience  de  combat  applicable  des  participants  pour  les  roles  qu’ils  etaient  appeles  a  jouer,  et 
(ii)  le  manque  de  stabilite  du  logiciel  de  base,  ce  qui  aurait  pu  se  traduire  par  de  la  frustration  chez  les 
utilisateurs  et  par  une  deviation  systematique  des  reponses. 
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Introduction 


This  report  details  the  purpose,  method,  and  results  of  a  Utility  Trial  of  the  Tactical  Battlefield 
Command  System  (TBCS)  /  Chameleon  software.  The  trial  was  conducted  by  Hum ansy stems'’ 
Incorporated  (HSI®)  under  original  contract  to  the  Defense  and  Civil  Institute  of  Environmental 
Medicine  (DCIEM),  now  Defence  Research  and  Development  Canada,  Toronto  (DRDC  Toronto)  on 
behalf  of  the  Director  of  Land  Requirements  4-5  (DLR  4-5)  in  partial  fulfilment  of  contract  #W771 1-6- 
7286/0 1-SRV. 

1.1.  Report  Format 

The  report  format  is  based  on,  and  customised  from,  MIL-STD-46855B  Human  Engineering 
Requirements  for  Military  Systems,  Equipment  and  Facilities  and  related  DID-DI-HFAC-80744 
Human  Engineering  Test  Report. 

1.2.  Purpose 

The  purpose  of  this  trial  was  to  provide  systematic,  useful  and  timely  user  feedback  to  the  design 
team,  in  order  to  enhance  the  utility  and  functionality  of  the  TBCS/Chameleon  system  and  to  assist 
with  the  direction  of  future  developments  of  the  system. 

1.3.  Objectives 

The  objectives  of  this  trial  were  to: 

•  Provide  user  feedback  on  the  utility  of  the  features  in  TBCS/Chameleon  as  they  apply  to 
various  combat  team  levels  of  command,  and 

•  Provide  user  feedback  on  the  ease  of  use  of  the  system’s  interface  and  functionality. 

1.4.  Equipment/Concept  Tested 

The  aim  of  the  TBCS/Chameleon  project  is  to  capture  requirements  for  the  development  of  a  Combat 
Team  Level  command  and  control  system.  The  concept  of  TBCS  is  to  provide  a  vehicle-based,  semi- 
automated  command  and  control  software  system  to  fit  within  the  series  of  battlefield  management 
systems:  Land  Force  Command  System  (LFCS)  at  the  Brigade  Group  level  and  Integrated  Personal 
Clothing  &  Equipment  (IPCE)  at  the  individual  soldier  level. 
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Version  3.1  of  TBCS/Chameleon  was  used  during  the  Utility  Trial.  Sections  of  each  of  the  key 
features  were  in  a  functional  state,  and  the  non- functional  components  were  demonstrated  at  the 
concept  level.  Future  versions  may  contain  more  features,  which  have  not  yet  been  developed  to  the 
point  where  they  can  be  included  in  the  software  as  non-functional  components.  These  were  reviewed 
at  the  concept  level  to  determine  initial  user  perception  of  utility.  The  key  areas  of  functionality 
available  for  this  utility  trial  included: 

•  Overlays 

•  Messaging 

•  Symbols 

•  Map  features 

•  Unit  Information 

•  TO&E 

•  Operation 

•  Status  Displays 

•  Vetronics 

•  System  options 

1.5.  Combat  Team  Tasks 

An  advance  to  contact,  Combat  Team  level  scenario  was  developed  to  allow  users  representing  various 
levels  of  command  (Sqn/Coy,  Trp/Pl)  to  perform  a  task  based  utility  evaluation  of  TBC  S/Chameleon. 
The  scenario  was  based  on  a  Janus  Simulator  exercise  used  by  the  Joint  Command  Staff  Training 
Centre  (JCSTC)  in  Kingston.  The  key  TBCS/Chameleon  tasks  users  performed  during  the  various 
segments  of  the  scenario  were: 

•  Contact  Report 

•  Location  Report 

•  Situation  Report  (Free  text  report) 

•  Plan  Creation  (battle  procedures,  drawing  a  trace  with  enemy  and  friendly  symbols, 
obstacles,  etc.) 

•  Overlay  Creation  (Hasty  Attack  Plan) 

•  Map  Navigation 

•  Unit  Familiarization  (TO&E  query,  resources,  etc.) 

The  tasks  listed  above  were  performed  throughout  the  scenario,  that  was  broken  into  several  segments 
to  allow  natural  break  points  for  user  evaluation.  These  included: 

1 .  Gather  background  unit/mission  information 

2.  Warning  Order 

3.  Operation  Order 

4.  Update  Warning  Order 

5.  Move  to  Assembly  Area 

6.  At  LD 

7.  Move  to  First  Objective 

8.  Contacts 

9.  Continue  Advance,  Parachute  Company  Delayed 

10.  Warning  Order  for  hasty  attack/defence 

1 1 .  Conduct  of  hasty  attack/defence 

12.  Consolidation 
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1.6.  Limitations 


The  number  of  participants  available  for  the  trial  limits  the  ability  to  generalise  the  data  and  to  perform 
inferential  statistical  analysis  of  the  results.  The  trial  participants  represent  a  cross  section  of  combat 
team  personnel,  who  each  have  potentially  very  different  needs  of  the  software.  Therefore,  when 
group  means  have  been  calculated,  we  have  been  careful  in  each  case  to  ensure  that  the  mean  was 
indeed  representative  of  the  group  as  a  whole,  and,  if  not,  to  point  out  individual  variations  in 
responses. 

The  use  of  a  single  scenario  may  be  a  limitation,  since  this  does  not  capture  the  full  range  of  mission 
contexts  and  tasks  associated  with  the  full  spectrum  of  combat  team  operations. 

Further  limitations  experienced  during  the  trial  will  be  discussed  in  the  method  and  discussion 
sections. 

1.7.  List  of  Acronyms 

AFV  Armoured  Fighting  Vehicle 

BG  Battle  Group 

BDE  Brigade 

BMS  Battlefield  Management  System 
CAPT  Captain 

CAS  EVAC  Casualty  Evacuation 
CF  Canadian  Forces 

CP  Command  Post 

C2  Command  &  Control 

CEOI  Communication  Electronic  Operating  Instructions 

CGI  Software  Developer 

CMBG  Canadian  Mechanized  Brigade  Group 

COY  Company 

CTA  Cognitive  Task  Analysis 

DCIEM  Defence  and  Civil  Institute  of  Environmental  medicine 
EN  Enemy 

FOO  Forward  Artillery  Observer 

FR  Friendly 

GPS  Global  Positioning  System 

G3  Operations  Staff  Officer 

G4  Logistics  Staff  Officer 

HCI  Human  Computer  Interface 

_ 1 1 S 1  1  luman.n-.yfe/m  Incorporated _ 
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IPCE 

Integrated  Personal  Clothing  &  Equipment 

JCSTC 

Joint  Command  Staff  Training  Center 

LAV 

Light  Armoured  Vehicle 

LT 

Lieutenant 

LD 

Line  of  Departure 

LFCS 

Land  Force  Command  System 

MAJ 

Major 

MASH 

Armour  Ammunition  Resupply  Report 

NATO 

North  Atlantic  Treaty  Organization 

0  Group 

Orders  Group 

OC 

Officer  Commanding 

ORBAT 

Order  of  Battle 

OVLAY 

Overlay 

PARA 

Parachute 

PC 

Personal  Computer 

PL 

Platoon 

PMO 

Project  Management  Office 

RCD 

The  Royal  Canadian  Dragoons 

3  RCR 

The  3rd  Battalion,  The  Royal  Canadian  Regiment 

RECCE 

Reconnaissance 

SA 

Situation  Awareness 

SGT 

Sergeant 

SQN 

Squadron 

TBCS 

Tactical  Battlefield  Command  System 

TO&E 

Technical  ORBAT  and  Equipment 

TOWUA  Tow  Under  Armour 

TP 

Troop 

WNG  0 

Warning  Order 

WO 

Warrant  Officer 

ZT 

Designated  Artillery  Target 

21/C 

Second  in  Command 
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1.8.  Applicable  Documents 

M1L-STD-46855B  Human  Engineering  Requirements  for  Military  Systems,  Equipment  and  Facilities; 
DID  DI-HFAC-80743  (Human  Engineering  Test  Plan);  and  DID  DI-HFAC-80744  (Human 
Engineering  Test  Report). 

Humansystems  Incorporated,  Preliminary  Cognitive  Task  Analysis  (CTA)  Conducted  With  Combat 
Team  Commanders,  Report  to  DCIEM,  January  1998. 
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Method 


2.1  Participants 

To  evaluate  the  utility  of  a  software  system  aimed  at  the  Combat  Team  level,  it  is  important  to  have 
representative  users  from  the  combat  team  involved  in  the  evaluation,  i.e.  trial  participants  should 
represent  a  cross  section  of  the  potential  user  population.  This  population  includes  (although  is  not 
limited  to)  combat  team  personnel  at  the  Squadron/Company  and  Troop/Platoon  level  from  the 
infantry,  armour,  artillery,  recce,  engineering  and  anti  armour.  The  table  below  shows  the  intended 
eight  member  cross  section  of  the  combat  team  for  the  advance  to  contact  scenario.  It  should  be  noted 
that  for  practical  and  logistical  reasons  some  users  would  be  required  to  “play”  more  than  one  role 
during  the  trial.  Also  shown  in  the  table  are  the  actual  characteristics  of  the  7  participants  who  were 
made  available  for  the  trial. 


Table  1:  Participant  Characteristics 


Requested  Participants  and  their  Characteristics 

Actual  Characteristics  of 
Trial  Participants 

Participant 

Representation/Experience  of 
Participant 

Rank 

Actual 

Rank 

Experience 

Requirement 

Met 

1. 

Engineering  Recce 

Troop  Leader  (Ell) 

Engineering  as  well  as  Recce 

Lt. 

Sgt 

Partial 

2. 

Armour  OC 

Sqn  comd  and  OC 

Maj. 

Capt 

Partial 

3. 

Infantry  OC 

Coy  comd  and  OC 

Maj 

n/a 

4. 

Recce  Troop  Leader 
(LAV  recce 
experience) 

Armour  Troop  Leader  as  well  as 
Recce 

Lt. 

Sgt 

No 

5. 

Platoon  Comd  (with 
anti  armour  experience) 

Platoon  Comd  as  well  as  anti 

armour 

Lt. 

Sgt 

No 

6. 

Armour  2i/c  (senior 
capt.  with  BC 
experience) 

Armour  comd  and  logistics  (should 
have  recce  experience  if  4  does 
not) 

Capt. 

Lt. 

No 

7. 

Infantry  2i/c 

Infantry  comd  and  logistics  (should 
have  anti  armour  experience  if  5 
does  not) 

Capt. 

Capt. 

Partial 

8. 

Artillery  Forward 
Observation  Officer 

FOO 

Lt./ 

Capt. 

WO 

Partial 

It  can  be  seen  from  table  1  that  there  was  a  significant  difference  between  the  desired  user  group  and 
the  actual  user  group  both  in  rank  and  experience.  This  affects  the  utility  evaluation  in  many  ways 
including: 

•  Lack  of  experience  in  all  aspects  of  the  position  means  that  the  participant  may  be  unable 
to  adequately  judge  the  utility  of  system  features. 

•  It  compromises  the  ability  to  role  play  non  current  position. 
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•  It  makes  it  more  difficult  for  users  to  separate  utility  from  ease  of  use  issues. 

•  It  reduces  the  ability  to  use  imagination  to  project  and  differentiate  TBCS  /  Chameleon 
use  to  different  segments  of  the  scenario. 


2.2  Schedule 

The  following  briefly  lists  the  schedule  of  events  for  the  four  day  trial: 


Day  1 

Day  2 

Day  3 

Day  4 

Morning 

Equipment/  Room 
set  up 

Participants  arrive 

TBCS/Evaluation 
introduction  to 
participants 

Participant  training 

Scenario  Play 
(Data  capture) 

Scenario  Play 

(Data  Capture) 

Afternoon 

Scenario  Testing 

Participant  training 
and  part  tasks 

Scenario 

Introduction/  Play 

Scenario  Play 
(Data  Capture) 

Focus  Group  and 
Wrap  up 

Discussion 

DAY  1 

This  was  primarily  intended  as  a  set  up  day  where  software  was  installed  and  the  appropriate  network 
connections  made.  The  set  up  took  place  in  the  morning  followed  by  machine  and  scenario  testing  in 
the  afternoon. 

DAY  2 

Participants  were  introduced  to  the  concept  of  TBCS,  the  trial  and  PMO  staff  and  the  purpose  and 
objectives  of  the  trial  during  the  morning  of  day  2.  For  the  duration  of  the  morning,  participants  were 
given  familiarisation  training  with  TBCS/Chameleon. 

Further  training  was  given  in  the  afternoon,  after  which  participants  performed  selected  part  tasks 
using  the  networked  PCs  and  Chameleon/TBCS  software.  Towards  the  end  of  the  afternoon 
participants  were  re-introduced  to  the  scenario  and  began  hands-on  scenario  play. 

DAY  3 

Each  stage  of  scenario  play  was  followed  by  a  questionnaire  and  focus  group  discussions.  This 
continued  until  the  scenario  was  completed  on  DAY  4 

DAY  4 

Scenario  play  and  data  capture  continued  until  mid  afternoon.  This  was  followed  by  a  final  focus 
group  where  key  issues  arising  from  the  trial  were  discussed.  Implementation  issues  and  future 
features  were  also  discussed. 

2.3  Physical  Layout  and  Equipment 

The  evaluation  was  conducted  in  a  large  room  with  enough  seating/computer/trial  administrator  space 
to  accommodate  approximately  16  people.  The  details  are  listed  below: 

•  a  large  classroom. 
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•  overhead  projector  and  computer  screen  projector, 

•  8  TBCS  computer  workstations  with  desks  and  tables. 

The  configuration  of  the  room  is  given  below: 


Observer 

seating 


T21/ 

62A 


131/ 

71F 


Ell/ 

62D 

T29 

Gil 

[ 

13  9  A 

_ 1 

T29A 

/ 


Overhead 
projection 
screen  / 
blackboard 


Admin 

19 


Figure  1:  Utility  trial  room  configuration. 

Note:  the  eight  rectangles  represent  the  combat  team  and  trial  administration  workstations  with  chairs 
and  networked  TBCS/Chameleon  laptops.  Please  note  the  trial  administration  workstation  was 
“Admin  /  9”. 

2.4  Call  signs 

The  call  signs  for  the  various  units  are  detailed  in  table  2  below. 


Table  2:  Call  sign  assignments 


Unit 

Call  sign 

Armoured  Engineer  Troop 

Ell 

Armour  OC 

T29 

Infantry  OC 

139 

Recce  Troop  Detachments 

62A,  62D 

Infantry  Platoon 

131 

Armour  2i/c 

T29A 

Infantry  2i/c 

13  9  A 

Artillery  Forward  Observation  Officer 

G1 

TOWUA 

71F 

2.5  Responsibilities 

The  responsibilities  for  the  TBCS  evaluation  were  shared  between  the  software  developers  (CGI),  the 
Project  Management  Office  (PMO)  and  Hum ansys terns  ’  Incorporated.  The  responsibilities  were  as 
follows: 
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TBCS  Introduction 

PMO 

TBCS  Familiarisation  and  Training 

PMO/HSI® 

Utility  Trial  Introduction 

HSI® 

Conduct  of  Scenario  Play 

HSI® 

Utility  Evaluation  Facility 

PMO 

Software  and  Computer  Equipment 

CGI/PMO 

Scenario  Script 

HSI® 

Pre -prepared  Computerized 

Scenario  Information 

CGI 

Presentation  Equipment 

PMO 

Questionnaire  Material 

HSI® 

Focus  Group  /  Wrap  Up  Discussion 
Presentation 

HSI® 

2.5.1  Training  of  Participants 

There  were  approximately  3  hours  of  familiarisation  training  with  TBCS/Chameleon  software,  starting 
with  an  oral  overview  of  the  system  features  with  no  participant  interaction  with  TBCS/Chameleon 
workstations.  Following  this  feature  based  training,  participants  completed  several  part  tasks  using 
their  workstations.  This  allowed  them  become  familiar  with  the  concepts  introduced  earlier  in  the 
training,  and  to  get  first-hand  experience  executing  tasks  with  the  system.  These  part  tasks  included: 

•  Map  navigation 

-  Find  units  (move  map) 

-  Zoom  in/out 

-  Change  re-enter  modes 

•  Create  and  send  a  free  text  and  log  rep  message 

•  Symbol  drawing 

-  Add  units 

-  Move  units 

-  Create  a  minefield 

-  Create  designated  artillery  target  (ZT) 

•  Create  and  send  a  order  with  trace 

•  Send  an  order  with  trace 

•  Receive  an  order  with  trace 
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2.6 


Procedure 


After  familiarisation  and  training,  participants  took  part  in  an  exercise  based  scenario  which  required 
them  to  complete  several  tasks  at  networked  workstations.  These  tasks  were  embedded  in  13 
segments  of  the  advance  to  contact  scenario.  Each  stage  required  the  use  of  a  range  of 
TBCS/Chameleon  functionality. 

2.6.1  Scenario  Description 

The  scenario  used  was  a  Combat  Team  advance  to  contact  level  scenario  from  an  exercise  used  by  the 
Joint  Command  Staff  Training  Centre  (JCSTC)  in  Kingston  for  the  JANUS  Combat  Team 
Commander  training  simulator  (see  Annex  A  for  full  details). 


Figure  2:  Initial  trace  showing  exercise  intent. 
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A  brief  summary  is  provided  here  (also  see  Figure  2).  2CMBG  is  to  secure  the  south  bank  of  the 
Petawawa  River  in  preparation  to  support  a  crossing  by  1 CMBG.  Three  phases  will  be  required  to 
secure  object  CAT  (Beachburg).  ( For  the  purpose  of  the  trial,  only  the  first  phase  was  relevant)  In 
Phase  1  2CMBG  will  advance  with  two  BG  up  with  RCD  left  and  3  RCR  right.  RCD  will  secure 
object  SNAKE  and  3  RCR  will  secure  object  BIRD  (Foresters  Falls).  Details  of  phase  1  are  for  PARA 
coy  combat  team  left  and  B  coy  combat  team  right  to  advance  with  O  coy  combat  team  in  reserve 
using  CLUB  route  etc.  ( The  trial  user  group  represented  B  coy  combat  team  on  the  right.)  The  full 
Operation  Order  can  be  found  in  Annex  B. 

Following  hands  on  training,  participants  were  instructed  to  take  positions  at  specific 
Chameleon/TBCS  networked  workstations,  which  comprised  the  various  elements  of  the  combat  team 
e.g.  T29, 139,  T29A,  I39A,  T21, 131,  Gl,  71F,  62A,  62D,  Ell.  Participants  were  re -briefed  as  to  the 
purpose  of  the  trial,  use  of  questionnaires  and  scenario  evaluation  framework.  Particular  attention  was 
given  to  instructing  users  to  “play  out”  the  scenario,  to  concentrate  on  providing  feedback  for  their 
assigned  position,  and  to  focus  on  the  utility  of  the  features  as  opposed  to  the  ease  of  use  of  the 
system.  As  some  participants  were  required  to  play  the  role  of  more  than  one  player,  they  were 
instructed  to  change  their  log-in  name  to  reflect  the  position  they  were  playing,  when  necessary,  e.g. 
for  sending  messages. 

Participants  used  a  custom  feature  of  the  software  to  load  pre-defmed  traces  that  display  different 
segments  of  the  advance  to  contact.  Every  trace  contained  the  basic  Cobden  map  (1:50  000,  Sheet  3 1 
F/10,  Edition  4)  and  some  details  such  as  map  position,  unit  positions,  trace  information,  and  enemy 
contacts  were  shown  on  each  of  these  scenarios.  After  loading  the  trace,  participants  received 
instruction  from  the  facilitators  (by  voice  or  using  TBCS/Chameleon  messaging)  or  from  other 
participants  e.g.  orders,  and  were  required  to  “play  out”  a  certain  part  of  the  scenario.  Instructions 
included  suggestions  to  participants  of  which  software  items  would  help  in  particular  task  executions. 

The  scenario  was  broken  into  segments  to  allow  natural  break  points  for  user  evaluation  (see  below). 
The  full  details  of  the  scenario  and  the  individual  events  and  tasks  that  were  planned  to  occur  can  be 
found  in  Annex  C.  Due  to  the  fact  that  participants  were  expected  to  “play”  during  the  scenario, 
flexibility  in  administration  and  timing  was  required  during  each  stage.  Where  software  items  or 
situations  were  not  covered  or  omitted  by  users  in  one  stage,  or  where  technical  difficulties  prevented 
task  completion,  the  scenario  was  modified  in  real-time  to  capture  these  in  later  segments. 

The  scenario  segments  were  as  follows, 

1.  Gather  background  unit/mission  information 

Participants  loaded  trace  1  containing  combat  team  positions  and  “0”  position,  a  paper  copy  of  the 
overall  intelligence  picture  (text  and  sketch)  and  instructions  for  the  software  use  and  tasks. 

2.  Warning  Order 

Using  trace  1,  participants  received  a  paper  copy  of  the  Warning  Order.  They  were  required  to  follow 
the  order  and  conduct  battle  procedures  appropriate  to  their  position. 

3.  Operation  Order 

Participants  loaded  trace  2  and  received  an  electronic  copy  of  an  Operations  order.  They  were  then 
required  to  follow  the  order  and  conduct  battle  procedures  appropriate  to  their  position. 

4.  Update  Own  Warning  Order 

Using  trace  2,  participants  were  instructed  to  make  final  preparations  as  necessary  in  preparation  for 
H-hr. 

5.  Move  to  Assembly  Area 

Participants  loaded  trace  3  and  moved  their  units  along  specified  routes  at  a  realistic  pace  to  the 
assembly  area  in  preparation  for  crossing  the  LD.  The  test  administrator  initiated  a  Situation  Report, 
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which  indicated  a  vehicle  hitting  a  mine  along  the  route;  this  required  messaging  among  the  combat 
team  to  maintain  traffic  flow. 

6.  At  Line  of  Departure  (LD) 

Using  trace  3,  the  combat  team  commanders  were  required  to  order  their  units  to  specific  positions  for 
crossing  the  LD  at  H-hr. 

7.  Move  to  First  Objective 

Using  trace  3,  the  combat  team  moved  each  of  their  own  units  across  the  LD  to  the  first  report  line, 
while  being  vigilant  for  enemy  forces. 

8.  Contacts 

Participants  loaded  trace  4  and,  while  advancing,  were  prompted  by  the  facilitators  (by  paper  or  voice) 
of  the  presence  of  enemy.  Users  were  expected  to  create,  send  and  receive  several  contact  reports  as  a 
result  of  prompting. 

9.  Continue  Advance,  PARA  Delayed 

Participants  loaded  trace  5.  Participants  used  the  appropriate  TBCS/Chameleon  tools  to  orient 
themselves  to  the  current  situation  on  the  battlefield.  The  left-hand  combat  team  was  shown  to  be 
delayed  by  messaging  and  by  map  indication. 

10.  Warning  Order  for  hasty  attack 

Participants  loaded  trace  6.  Sufficient  contacts  were  indicated  to  demonstrate  the  need  for  a  Hasty 
Attack  at  Millars  Comers.  The  combat  team  commanders  were  prompted  to  create  and  distribute  a 
hasty  attack  plan  as  an  electronic  text  and  electronic  sketch  Other  members  of  the  combat  team 
received  this  order  and  carried  out  the  necessary  steps  to  prepare  to  execute  it. 

11.  Warning  Order  for  hasty  defence 

Participants  loaded  trace  7  containing  their  combat  teams  positions  for  the  conduct  of  the  hasty  attack. 
The  combat  team  commanders  were  alerted  by  a  Situation  Report  about  an  enemy  advance,  this 
prompted  the  need  for  the  planning  and  conduct  of  a  hasty  defence  using  the  software. 

12.  Conduct  of  hasty  defence  /  Conduct  of  attack 

Participants  loaded  trace  8  showing  the  results  of  the  defence  and  were  required  to  move  into  position 
to  execute  the  hasty  attack.  Participants  were  prompted  about  the  features  of  TBCS  that  could  assist 
them  in  the  conduct  of  an  attack. 

13.  Consolidation 

Participants  were  required  to  conduct  consolidation  activities  using  trace  8. 


In  summary,  the  key  tasks  expected  of  the  user  during  the  various  segments  of  the  scenario  were: 

•  Unit  Familiarization  (TO&E  query,  resources,  etc.) 

•  Contact  Report 

•  Location  Report 

•  Situation  Report  (Free  text  report) 

•  Plan  Creation  (Drawing  a  trace,  enemy  and  friendly  symbols,  obstacles,  etc.) 

•  Overlay  Creation  (Hasty  Attack  Plan) 

•  Map  Navigation 


2.6.2  Data  Recording 

During  scenario  segments  data  were  captured  through  questionnaires  and  direct  observation. 
Following  each  segment  of  the  scenario,  participants  were  asked  to  complete  a  questionnaire  and  then 
participate  in  a  focus  group  discussion  of  the  key  components  used  in  the  segment.  After  the  final 
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scenario  segment,  participants  participated  in  a  final  focus  group  session  to  review  TBCS/Chameleon 
on  key  issues  relating  to  utility,  ease-of-use  and  problem  areas. 

2. 6.2.1  Questionnaires 

A  task  based  questionnaire  was  administered  after  completion  of  each  scenario  segment  (see  Annex  C) 
to  elicit  information  about  how  well  the  TBCS/Chameleon  system  allowed  users  to  perform  identified 
tasks  that  were  done  during  each  mission  segment.  The  questionnaire  focused  on  three  areas  of  utility 
and  two  aspects  of  ease  of  use  of  the  major  TBCS/Chameleon  functions.  The  utility  questions 
addressed  the  overall  usefulness  of  the  feature  in  question,  how  it  would  impact  on  operational 
effectiveness  and  whether  it  represented  an  improvement  over  current  capabilities.  The  ease  of  use 
questions  focussed  on  ease  of  use  in  the  trial  setting  and  projected  ease  of  use  for  typical  field 
conditions. 

2. 6.2.2  Direct  Observation 

Test  personnel  also  observed  participants  as  they  completed  the  various  scenario  segments  and 
recorded  ongoing  participants’  comments.  The  observations  and  information  gathered  served  to 
prompt  the  discussion  in  the  focus  group  debriefings  at  the  end  of  each  segment. 

2. 6.2. 3  Focus  Group  De-briefings 

Following  each  scenario  segment,  structured  open-ended  questions  were  asked  by  the  trial 
administrators.  The  specific  questions  evolved  as  the  scenarios  were  played  out  and  reflected  areas  of 
TBCS/Chameleon  that  participants  appeared  to  have  had  significant  problems  or  felt  were  particularly 
easy/useful.  Participants  were  also  encouraged  to  discuss  any  other  aspect  of  the  system  they  felt 
relevant. 
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3 


Results 


3.1  Outline 

The  results  of  the  utility  trial  are  presented  in  this  section.  For  each  major  section  of 
TBCS/Chameleon  functionality  participants  mean  ratings  are  followed  by  integrated  participant 
comments  and  F1F  team  observations. 

3.2  Reliability  and  Validity 

Before  analysing  the  results  of  the  trial,  we  briefly  review  some  important  issues  concerning  the 
reliability  and  validity  of  the  data  obtained.  The  concepts  of  reliability  and  validity  are  central  to  most 
scientific  research  undertakings  and  must  also  be  applied  to  evaluations  and  field  assessments.  In 
research  involving  human  performance,  care  in  the  design  of  studies  becomes  the  most  critical  element 
in  ensuring  that  the  study  achieves  its  objectives.  The  numbers  generated  by  poorly  designed  studies 
look  no  different  from  those  obtained  from  well-designed  studies.  The  key  is  to  ensure  that  the 
numbers  reflect  the  measurement  objectives  of  the  study,  rather  than  being  the  result  of  intrusions  of 
artefacts  and  unstable  measuring  instruments. 

3.2.1  Reliability 

The  underlying  principle  of  reliability  concerns  the  stability  of  the  data  and  the  confidence  that  one 
has  that  it  is  representative  of  the  behaviour  in  question.  The  key  to  establishing  reliability  is 
repetition.  That  is,  if  the  evaluation  were  to  be  repeated  on  successive  occasions  the  same  or  similar 
outcomes  would  ensue. 

Critical  factors  that  influence  reliability  in  the  context  of  most  field  evaluations  are  outlined  below 
together  with  a  brief  assessment  of  how  well  the  objective  of  each  factor  was  achieved  during  the 
evaluation  trial. 

•  The  precision  with  which  procedures  are  defined  prior  to  testing:  a  poorly  considered  trial 
script  will  allow  significant  variations  to  creep  in  each  time  the  scenario  is  conducted.  For 
example,  if  explicit  instruction  concerning  how  subjects  are  to  perform  a  task  are  not 
given,  then  high  variability  in  the  way  subjects  approach  the  task  may  occur.  (Comment: 
these  aspects  were  adequately  controlled  during  the  trial,  given  that  some  flexibility  and 
range  of  behaviours  was  desired) 

•  The  extent  to  which  independent  variables  are  tightly  controlled.  {Comment:  factors  such 
as  unreliability  of  the  software  and  network  served  to  undermine  reliability) 

•  The  consistency  in  application  of  the  script  by  test  administrators  (Comment:  this  was 
reasonably  well  controlled  during  the  trial) 

•  The  extent  to  which  intervening  variables,  which  could  influence  test  outcome,  are 
anticipated  and  controlled.  (Comment:  intrusive  comments  and  intervening  behaviours  by 
trial  observers  were  clearly  a  problem  on  several  occasions  during  the  trial). 

•  The  collection  of  a  sufficient  number  of  data  points  to  ensure  that  the  performance  of 
interest  is  sampled  with  adequate  precision.  (Comment:  across  the  entire  combat  team  an 
adequate  number  of  data  points  were  collected:  however,  since  only  one  individual 
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provided  one  data  point  for  each  of  the  system  features  evaluated,  the  reliability  is  low  in 
terms  of  any  analysis  for  individual  combat  team  positions.) 

•  Consideration  and  control  of  temporal  order  effects  which  may  influence  performance,  for 
example,  increasing  familiarity  with  the  task,  or  the  converse  increasing 
boredom.  ( Comment:  because  of  time,  budgetary >  and  logistical  constraints,  it  was  not 
possible  to  control  for  potential  order  effects). 

Overall,  in  the  case  of  the  present  evaluation,  practical  constraints  on  the  sample  size  and  the  time 
available  to  conduct  the  trial  are  major  concerns  for  the  reliability  of  the  data  obtained.  Hence  the 
results  of  the  trial  should  be  seen  as  suggestive  rather  than  definitive,  particularly  from  the  point  of 
view  of  trying  to  capture  potential  differences  in  evaluation  across  different  combat  team  positions. 

3.2.2  Validity 

The  concept  of  validity  deals  with  whether  the  measurements  obtained  are  consistent  with  principle  or 
evidence.  Thus,  an  experiment  or  evaluation  can  be  reliable,  but  its  conclusions  invalid  for  a  host  of 
reasons.  The  reverse  is  not  true,  however,  an  evaluation  could  not  be  valid  if  the  results  were 
unreliable.  Of  primary  concern  is  the  issue  of  construct  validity,  which  can  be  characterised  as  the 
development  of  sound  operational  measures  for  the  material  or  concepts  being  studied.  Good 
construct  validity  means  that  we  measure  that,  and  only  that,  we  want  to  be  measuring.  The  key  to 
controlling  threats  to  construct  validity  is  to  be  aware  and  continuously  vigilant  of  variables  that  may 
contaminate  an  evaluation.  In  the  case  of  the  present  trial,  we  have  attempted  to  increase  construct 
validity  by  addressing  three  separate  aspects  of  system  utility,  rather  than  using  just  a  single  measure. 
Further,  we  attempt  to  increase  validity  by  measuring  usability  issues  separately  from  utility  issues. 
Annex  D  provides  a  more  detailed  analysis  of  issues  relating  to  the  validity  of  the  trial. 

3.2.3  Data  Quality 

One  final  aspect  of  the  data  to  be  considered  is  the  extent  of  missing  or  questionable  data.  The  whole 
data  set  comprised  twenty  five  system  features  rated  on  five  questions  by  the  seven  trial  participants  to 
yield  a  potential  total  of  875  data  points.  Over  this  entire  set,  there  were  eleven  missing  data  points; 
these  resulted  from  the  Engineering  Recce  and  Battle  Captain  each  failing  to  rate  one  system  feature 
entirely,  in  addition,  the  Battle  Captain  failed  to  provide  a  rating  for  two  other  questions.  There  was 
no  consistent  evidence  of  questionable  data,  for  example  where  a  trial  participant  just  provides  the 
same  rating  for  every  question.  However  when  the  entire  data  set  is  examined  across  different  combat 
team  positions,  the  individuals  playing  the  Battle  Captain  and  Engineering  frequently  generated 
ratings  which  were  inconsistent  with  the  other  trial  participants. 

3.3  Participant  Characteristics 

The  seven  participants’  mean  experience  in  the  Canadian  Forces  was  13.5  years.  There  was  a 
discrepancy  between  the  characteristics  of  the  requested  and  actual  user  group  both  in  rank  and 
experience  (see  Method  section  Table  1). 

In  general,  participants  reported  having  good  familiarity  with  computers.  When  asked  about  the 
frequency  of  use  of  computer  related  issues,  participants  reported  frequent  use  of  keyboards,  desktop 
PCs  and  Windows  95,  and  between  occasional  and  frequent  use  of  Laptop  PC  and  Windows  3.1.  The 
subjects  also  reported  infrequent  use  (between  never  and  occasional)  of  Mac/Apple  products. 
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3.4  Network  system  stability 

For  most  TBCS/Chameleon  operations  performed  at  individual  participant  workstations  in  stand-alone 
mode,  the  software  performed  as  expected.  Participants  explored  much  of  the  software  and  tried  many 
features  that  worked  or  did  not  work.  The  non-functioning  feature  gave  a  message  saying  it  was  not 
available  in  the  current  system,  or  required  an  explanation  from  the  trial  administrators,  or  revealed  an 
unanticipated  bug  in  the  software. 

The  networked  TBCS/Chameleon  system  was  not  as  stable  as  expected  and  resulted  in  several  system 
crashes.  Crashes  occurred  more  frequently  with  increased  message  traffic,  and  as  a  result  the  pace  of 
the  trial  was  slowed.  The  software  developers  (CGI)  were  monitoring  when  and  where  system  crashes 
occurred  and  their  cause  will  be  reported  in  their  after  action  report.  It  should  be  noted  that  the  crashes 
in  the  system  did  not  affect  the  utility  evaluation  of  the  TBCS  and  that  all  the  participants  were  briefed 
that  the  aim  of  the  trial  was  to  test  the  utility  of  the  information  provided  and  not  to  focus  on  the 
reliability  of  the  program. 

3.5  Trial  Controls 

Conditions  of  the  trial  were  controlled  to  the  highest  extent  possible  with  the  exception  of  observers. 
The  presence  of  software  personnel  (2  people)  was  required  in  order  to  maintain  a  functioning 
network.  However,  at  one  point  during  the  trial  there  were  6  observers,  2  software  personnel  and  2 
trial  administrators  and  only  7  trial  participants.  The  observers  included  Majors  from  DCIEM  and  the 
trial  units  as  well  as  Captains  from  the  PMO  and  trials  units.  The  presence  of  observers  can  have 
several  effects  on  trial  participants  and  ultimately  has  the  potential  to  affect  the  overall  results  of  the 
trial.  Simply  by  their  presence,  observers  put  pressure  on  participants  to  behave  or  perform  in  a 
manner  they  would  not,  had  the  observer  not  been  there.  Second,  some  observers  cannot  help  but  get 
involved  in  the  trial  and  interact  with  participants.  This  can  interrupt  the  scenario,  waste  time  and 
stem  the  flow  of  data  from  the  users.  For  the  most  part,  observers  did  not  take  part  in  the  latter  activity 
but  no  doubt  affected  the  participants  by  their  presence. 

3.6  Analysis  of  Results 

In  this  section  of  the  report,  a  narrative  description  of  the  utility  data  integrated  with  the  usability  data 
is  provided  on  a  feature  by  feature  basis.  The  questionnaire  data  are  presented  as  mean  rating  scores 
for  each  individual  question  for  each  of  the  major  features  of  the  system,  and  are  presented  in  the  form 
of  inset  thumbnail  graphs.  In  addition  to  the  three  individual  utility  questions,  a  composite  utility 
rating  based  on  the  mean  of  these  three  questions  was  calculated. 

In  the  cases  where  there  are  deviations  in  individual  ratings  from  the  overall  group  tendency,  such 
differences  across  the  seven  positions  are  noted  for  the  utility  data  only.  For  each  of  the  system 
functions,  the  numerical  questionnaire  data  are  integrated  with  a  summary  of  the  comments  provided 
by  the  test  participants. 
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As  a  guide  to  interpreting  the  numerical  data,  the  following  conventions  have  been  adopted. 

UTILITY 


<3 

=  less  than  acceptable 

3-3.9 

=  moderate  utility 

•  4-5 

USABILITY 

=  strong  utility. 

1-1.9 

=  clearly  unacceptable 

2-2.8 

=  unacceptable 

2. 8-3.2 

=  marginally  acceptable 

3.2-3. 9 

=  acceptable. 

4-5 

=  clearly  acceptable 

3.6.1.  Overlays 

3.6.1. 1  Overlays:  control  measures 

Terminology  in  the  overlay  section  seemed  to  cause  confusion  among  users.  Some  of  the  terms 
seemed  to  be  “computer  speak”  and  others  were  part  military/part  computer.  The  task  flow  also 
caused  some  confusion  i.e.  tasks  available  under  the  overlay  buttons  were  not  expected  to  be  there  by 
users. 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  borderline  in  terms  of  allowing  them  to  operate 
effectively  (3.0). 

2.  Was  moderately  desirable  (3.5). 

3.  Was  about  the  same  as  the  current  capability  (2.9). 

4.  Was  of  moderate  utility  (3.1). 

5.  Usability  in  the  test  bed  was  unacceptable  (2.6). 

6.  Usability  in  the  field  would  be  clearly  unacceptable 

(2.1). 

Ratings  on  question  1  were  inconsistent  across  the 
group;  the  feature  received  strong  ratings  from  the  FOO,  the  2i/c  Infantry  and  the  Combat  Team 
Commander,  but  ratings  of  “1”  from  the  Engineering/Recce  and  the  Battle  Captain.  Ratings  were  also 
inconsistent  on  question  2  where  the  feature  was  rated  somewhat  better  by  the  FOO  and  the  2i/c 
Infantry,  but  worse  by  the  Platoon  Commander. 

•  Participant  Comments 

The  ability  to  toggle  on  and  off  the  various  layers  was  seen  as  very  useful  as  it  could  clear  up  clutter 
and  detail  could  be  quickly  accessed. 

3.6.1.2  Co-ordinate  plans 

Ratings  suggest  that  users  feel  that  this  feature: 


1.0  2.0  3.0  4.0  5.0 


Operate  Effectively 


Compared  with  now 

Mean  Utility 

Usability  Now 

Usability  in  the  field 
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1.  Was  somewhat  helpful  in  terms  of  allowing 
them  to  operate  effectively  (3.1). 

2.  Was  a  moderately  desirable  feature  (3.4). 

3.  Was  slightly  better  than  the  current  capability 
(3.3). 

4.  Was  of  moderate  utility  (3.3). 

5.  Usability  in  the  test  bed  was  unacceptable  (2.6). 

6.  Usability  in  the  field  would  be  unacceptable 

(2.6). 


1.0  2.0  3.0  4.0  5.0 


On  questions  1  and  2  the  Engineering/Recce 

provided  much  lower  ratings  than  the  other  combat  team  members. 


•  Participant  Comments 

The  FOO  found  these  features  to  be  especially  useful  but  as  with  others  had  difficulty  with  the  ease  of 
use. 


The  Coy  2i/c  stated  that  flanking  unit  information  might  be  useful  here  i.e.  the  ability  to  turn  on 
flanking  units  could  help  planning. 


3.6.1.3  Orders 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  not  very  helpful  in  terms  of  allowing  them 
to  operate  effectively  (2.6). 

2.  Was  a  moderately  desirable  feature  (3.3). 

3.  Was  about  the  same  as  the  current  capability 
(3.1). 

4.  Was  of  moderate  utility  (3.0). 

5.  Usability  in  the  test  bed  was  unacceptable  (2.0). 

6.  Usability  in  the  field  would  be  clearly 
unacceptable  (1.9). 


1.0  2.0  3.0  4.0  5.0 


This  feature  received  inconsistent  ratings  across  the  group  on  all  three  utility  questions.  For  question  1, 
the  feature  was  generally  not  seen  as  allowing  the  user  to  operate  effectively,  with  the  exception  of  the 
Combat  Team  Commander  who  gave  the  feature  a  five  rating.  On  question  3,  this  feature  was  not 
rated  as  being  an  improvement  over  the  current  capability,  except  for  the  Combat  Team  Commander, 
who  again  gave  a  five  rating.  The  consistently  low  ratings  for  usability  suggest  that  the  feature  is 
somewhat  difficult  to  use  in  its  present  format. 

•  Participant  Comments 

Users  had  major  problems  using  orders.  “This  feature  seems  time  consuming  and  complicated  to  use 
and  may  take  time  away  from  battle  procedure  at  higher  levels.”  (Note  the  representatives  of  Trp  and 
PI  were  not  able  to  state  whether  or  not  they  should  have  this  feature).  The  other  big  issue  noted  by 
users  was  that  they  did  not  want  an  order  to  take  the  place  of  an  O  Group.  They  said  this  would  be  a 
useful  compliment  to  existing  procedures  but  would  not  allow  the  commander  to  express  command 
intent  or  feel  confident  that  subordinates  understand  the  plan. 
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A  recurring  issue  is  seeing  text  orders  and  the  map  at  the  same  time.  Users  stated  that  the  text  order 
should  be  at  the  side  of  the  map  and  trace  or  appear  in  a  resizable  window  allowing  both  to  be 
efficiently  viewed  at  the  same  time. 

3.6.2  Messaging 

The  messaging  aspect  of  TBCS  is  a  very  powerful  and  complex  feature,  not  all  aspects  of  which  could 
be  assessed  in  the  utility  trial.  The  evaluation  concentrated  upon  the  overall  utility  of  the  feature  as 
well  as  three  specific  forms  commonly  used,  namely,  contact  reports,  warning  orders  and  sending  an 
overlay. 

3. 6.2. 1  Messaging  general 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to  operate 
effectively  (4.0). 

2.  Was  a  desirable  feature  (4.3). 

3.  Was  about  the  same  as  than  the  current  capability 
(3.0). 

4.  Was  of  moderate  utility  (3.8). 

5.  Usability  in  the  test  bed  was  acceptable  (3.6). 

6.  Usability  in  the  field  would  be  marginally 
acceptable  (3.0). 

On  question  3  opinions  were  divided,  with  the  Engineering/Recce,  FOO  and  the  2i/c  Infantry  rating  it 
as  somewhat  better  than  now,  and  the  Battle  Captain  and  Platoon  Commander  rating  it  as  worse  than 
now.  As  presently  configured,  this  feature  provides  acceptable  usability  both  for  desk  based  and  field 
operations. 

•  Participant  Comments 

There  was  consensus  among  users  that  time  to  make  reports  should  be  minimised,  information  clutter 
should  be  reduced  and  message  information  needs  to  be  integrated  with  map  information  (requiring 
simultaneous  access  to  messaging  text  and  map  display/navigation).  Users  commented  that  creating 
and  sending  messages  could  be  streamlined  by  taking  advantage  of  the  automation  available  (pre¬ 
formatting  and  default  recipient  lists).  Loc  and  move  reps  should  not  be  automatically  sent  on 
movement,  as  GPS  will  indicate  position  on  map.  This  information  currently  fills  up  in  baskets  too 
quickly.  Users  expressed  concern  that  they  would  be  spending  too  much  time  sorting  through  mail 
instead  of  commanding  vehicle/troops.  When  deleting  messages,  users  could  only  select  one  message 
at  a  time  and  commented  that  multiple  message  selection  would  be  helpful. 

During  focus  groups  participants  commented  that  the  “preferred  reports”  message  feature  was  useful. 
They  thought  this  provided  them  with  the  flexibility  they  need  to  customize  the  message  display  to 
access  the  most  frequently  used  messages  more  frequently. 

The  overall  comments  following  administration  of  the  “overall”  questionnaire  were  that  speed  and 
ease  of  creating,  sending  and  receiving  messages  must  be  worked  on.  This  will  enhance  the  ease  of 
use  of  a  potential  useful  tool.  One  user  suggested  that  there  needs  to  be  standardisation  of  message 
formats  across  all  arms  to  fully  use  this  system.  Users  suggested  that  they  should  be  able  to  see  map 
and  messages  at  the  same  time,  indicate  new  messages  more  prominently,  and  need  more  pre-set 
features. 
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3. 6.2.2  Contact  reports 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to  operate 
effectively  (3.9). 

2.  Was  a  desirable  feature  (4.1). 

3.  Was  worse  than  the  current  capability  (2.3). 

4.  Was  of  moderate  utility  (3.4). 

5.  Usability  in  the  test  bed  was  marginally  acceptable 
(2.9). 

6.  Usability  in  the  field  would  be  unacceptable  (2.3). 


•  Participant  Comments 

Although  users  saw  high  potential  utility  for  the  contact  report,  they  found  the  contact  report 
somewhat  cumbersome  and  time  consuming.  Major  benefits  were  seen  with  this  feature  if  the 
following  provisions  are  implemented: 

•  integration  with  a  laser  range  finder, 

•  indication  of  off  screen  contacts, 

•  rapid  situation  awareness  of  direction  of  movement  (visual  indicator), 

•  contact  wait  out  feature  (one  click  for  contact  wait  out  and  possible  one  click  to  indicate 
position  or  direction  of  contact) 

•  new  contact  reports  should  flash  on  screen 

•  any  information  about  the  contact  should  be  available  by  clicking  on  the  vehicle  on  the 
map, 

•  indication  if  the  contact  has  been  destroyed  (one  user  suggested  greying  out  the  symbol; 
question  as  to  who  is  allowed  to  indicate  this), 

•  a  pick  list  of  vehicle  set  up  for  the  expected  enemy  Orbat 

Some  users  still  felt  that  the  contact  report  should  not  and  will  not  ever  replace  voice  contact  reports. 


1.0  2.0  3.0  4.0  5.0 


Hummsystems 


TBSC/Chameleon  Utility  Trial  Report 


Page  20 


3. 6.2. 3  Warning  order 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to 
operate  effectively  (4.0). 

2.  Was  a  highly  desirable  feature  (4.6). 

3.  Was  better  than  the  current  capability  (4.0). 

4.  Was  of  strong  utility  (4.2). 

5.  Usability  in  the  test  bed  was  acceptable  (3.9). 

6.  Usability  in  the  field  would  be  acceptable  (3.3). 


This  feature  received  consistently  high  ratings  for 
all  utility  questions  from  all  except  the  2i/c  Infantry  and  the  Troop  Leader,  who  did  not  rate  the  feature 
as  allowing  them  to  operate  effectively. 

•  Participant  Comments 

Improvements  to  this  highly  useful  feature  mainly  concerned  screen  space.  Users  could  not  relate  the 
text  message  (order,  routes,  etc)  to  the  map  and  wrote  down  much  of  the  information  and  then  related 
it  to  the  map.  The  forwarding  feature  is  good  (“. .  .will  eliminate  inaccuracies  of  speech  and  be  so 
much  faster. . .”)  as  long  as  the  order  can  be  edited.  However  it  is  too  slow  in  current  implementation 

i.e.  copy  and  paste  into  each  section  is  cumbersome.  Some  users  suggested  that  a  hard  copy  print  out 
would  be  better  if  the  display  format  will  not  support  efficient  viewing  of  the  map  and  the  order. 

The  comment  was  made  that  the  user  should  be  able  to  select  a  default  group  in  the  recipient  list. 

The  “Orders  in  minutes”  alert  should  indicate  when  the  orders  will  be  delivered  in  case  you  do  not 
receive  the  order  exactly  when  it  is  sent. 

One  user  commented  that  it  is  critical  that  the  warning  order  (and  other  orders  such  as  hasty  attack)  be 
acknowledged  i.e.  it  is  not  enough  to  just  send  a  reply  that  the  message  has  been  received  but  that  it 
has  been  read  and  understood. 

3. 6.2. 4  Send  overlay 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to 
operate  effectively  (3.7). 

2.  Was  a  desirable  feature  (4.3). 

3.  Was  a  little  better  than  the  current  capability 
(3.3). 

4.  Was  of  moderate  utility  (3.8). 

5.  Usability  in  the  test  bed  was  marginally 
acceptable  (2.9). 

6.  Usability  in  the  field  would  be  marginally 
acceptable  (3.0). 

The  Engineering/Recce  and  the  Platoon  Commander,  who  rated  the  feature  as  being  somewhat  worse 
than  the  existing  capability,  provided  the  only  negative  ratings. 
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•  Participant  Comments 

The  concept  of  this  feature  appears  to  be  more  useful  than  the  ratings  indicate.  That  is,  users 
commented  that  sending  a  sketch  with  a  warning  order  would  be  useful  as  everyone  would  have  the 
same  information  and  radio  traffic  would  be  reduced.  However  in  its  current  implementation,  users 
found  this  process  of  creating,  sending  and  receiving,  as  with  orders,  to  be  far  too  complicated.  Users 
said  they  would  use  this  feature  for  a  deliberate  attack  but  not  for  a  hasty  attack  as  it  would  take  longer 
than  a  radio  order.  One  user  commented  that  for  the  OC  and  above  it  would  be  a  useful  feature  for 
deliberate  attack  and  similar  planning  but  is  too  time  consuming  otherwise.  Another  user  said  sending 
an  overlay  is,  “. . .  a  desirable  feature  however  too  time  consuming  and  rigid.  It  needs. .  .quick  freehand 
drawing. . .  so  as  not  to  steal  time  from  the  battle.” 

Engineering  commented  that  this  feature  would  receive  little  use.  Artillery  said  they  need  the  ability 
to  enter  target  numbers  on  the  overlays. 

Comments  made  at  the  end  of  the  trial  in  the  overall  section  rating  section  of  the  questionnaire  were 
unanimous  : 

•  the  concept  of  sending  an  overlay  is  good, 

•  many  hours  will  be  saved  (no  more  trace  copying), 

•  overlays  must  be  able  to  be  sent  quickly, 

•  users  need  more  flexibility  in  drawing  tools. 

3.6.3  Symbols 

3.6.3. 1  Friendly  units  Orbat 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to  operate 
effectively  (3.7). 

2.  Was  a  desirable  feature  (3.7). 

3.  Was  better  than  the  current  capability  (4.1). 

4.  Was  of  moderate  utility  (3.9). 

5.  Usability  in  the  test  bed  was  acceptable  (3.6). 

6.  Usability  in  the  field  would  be  marginally 
acceptable  (3.0). 

•  Participant  Comments 

Users  had  some  confusion  between  the  “ideal”  Orbat  under  TO&E  and  the  Orbat  under  symbols. 

They  wanted  much  of  the  information  under  TO&E  resources  to  be  current  information  for  their  own 
combat  team  or  unit,  so  they  could  use  one  feature  for  both  planning  and  drawing. 

Most  users  commented  that  this  feature  has  potential  to  be  very  useful  for  a  planning  tool  for  Sqn/Coy 
level  commanders  but  would  not  generally  be  used  below  this  level.  Suggested  improvements  to  the 
utility  of  this  feature  were  to  include  the  status  of  each  unit  such  as  vehicle  and  weapon  status,  weapon 
symbol,  weapon  range,  daily  battle  loss. 
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Users  also  commented  that  the  status  of  a  unit  could  be  graphically  displayed  here  to  give  the 
commander  a  quick  summary  of  the  capability  of  the  unit.  The  commander  could  then  “expand”  the 
Orbat  at  will  to  see  further  detail  and  status/capability  information. 

With  regard  to  friendly  symbols  on  the  map,  users  commented  that  the  option  to  filter  units  and  show 
different  aggregations  was  good  but  must  be  simple  to  use.  Also  the  ability  to  select  how  units  are 
show  e.g.  weapons,  Tac  or  call  sign,  is  very  useful. 

The  video  clip  feature  was  seen  as  having  considerable  potential.  Users  suggested  that  the  window  be 
resizable  to  enhance  its  utility. 

Overall  comments  were  favourable,  with  one  user  stating  that  he  preferred  to  see  call  signs  on  the  map 
instead  of  Tac  signs.  This  indicates  that  the  options  are  a  good  idea.  However,  improvements  could 
be  made  by  “clicking”  on  a  unit  for  a  small  identification  information  box  rather  than  it  popping  up  by 
itself  with  information  useful  for  logistics  only. 


3. 6.3. 2  Enemy  units  editor 

Ratings  suggest  that  users  feel  that  this  feature: 

1 .  Was  somewhat  helpful  in  terms  of  allowing  them 
to  operate  effectively  (3.3). 

2.  Was  a  desirable  feature  (3.7). 

3.  Was  somewhat  worse  than  the  current  capability 
(2.7). 

4.  Was  of  moderate  utility  (3.2). 

5.  Usability  in  the  test  bed  was  marginally 
acceptable  (3.2). 

6.  Usability  in  the  field  would  be  marginally 
acceptable  (2.8). 
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•  Participant  Comments 

This  feature  was  seen  as  having  low  utility  for  combat  operations.  Users  said  that  it  would  be  ,  “Good 
for  planning  or  in  the  intelligence  shop  or  CP,  I  don’t  think  I  would  use  it  in  the  field.”.  Users  said 
that  symbols  should  be  created  and  sent  from  “higher”  and  just  allow  them  to  pick  from  the  list  in 
contact  reports  and  from  the  editor.  A  suggestion  was  made  to  include  dismounted  enemy  units. 

3.6.4  Map  functions 

Three  aspects  of  the  map  and  related  capabilities  were  evaluated: 

•  the  basic  map  format  which  included  aspects  such  as  scale  and  detail 

•  map  navigation,  that  is  the  user's  ability  to  move  around  within  the  current  window  and 
between  the  presently  viewed  section  of  the  map  and  other  areas  currently  lying  outside  of 
the  present  window 

•  map  drawing  and  annotation,  which  involves  such  tasks  as  drawing  lines,  placing 
symbology. 
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3.6.4. 1  Map  format 
•  Questionnaire  Ratings 

Ratings  suggest  that  users  feel  that  the  map  format: 

1.  Was  somewhat  helpful  in  terms  of  allowing  them  to 
operate  effectively  (3.3). 

2.  Was  a  desirable  feature  (4.3). 

3.  Was  somewhat  worse  than  the  current  capability 

(2.8). 

4.  Would  be  useful  (3.4). 

5.  Usability  in  the  test  bed  was  unacceptable  (2.5). 

6.  Usability  in  the  field  would  be  unacceptable  (2.6). 


1.0  2.0  3.0  4.0  5.0 


Responses  for  items  1  and  2  varied  considerably  between  participants.  In  particular,  on  question  1  both 
the  Combat  Team  Commander  and  the  2i/c  Infantry  differed  from  the  other  five  participants  and  rated 
this  feature  as  being  somewhat  ineffective.  On  question  3  users  were  even  more  divided  over  whether 
this  feature  was  seen  as  being  an  improvement  over  the  current  capability.  Both  the  Combat  Team 
Commander  and  the  Engineering/Recce  rated  this  as  being  much  better  than  now,  the  Troop  Leader 
about  the  same  as  now,  and  the  remaining  four  positions  as  being  worse  to  much  worse  than  now. 
These  lower  ratings  may  have  been  due  to  contamination  of  this  utility  rating  by  usability  concerns, 
since  both  for  the  test  use  and  anticipated  usability  in  the  field,  ratings  were  less  than  acceptable 

•  Participant  Comments 

Comments  concerning  map  format  were  lengthier  and  more  insightful  than  for  any  other  feature. 

Some  users  (senior  combat  team  personnel)  expressed  general  concerns  over  the  size  of  the  map 
display.  They  had  difficult  imagining  how  they  would  function  with  the  limited  view  i.e.  being  able  to 
zoom  in  sufficiently  to  see  detail  while  still  trying  to  maintain  the  “larger  picture”.  These  users 
suggested  they  would  rely  on  a  paper  map  essentially  because  the  utility  of  the  current  implementation 
of  the  map  on  a  laptop  size  display  was  too  small. 

Some  users  were  concerned  with  the  lack  of  sufficient  detail  on  the  map  due  to  the  resolution  of  the 
screen,  and  suggested  that  the  paper  map  would  be  more  useful  for  infantry.  One  user  said  the  colour 
between  the  trees  and  water  was  too  close.  At  the  same  time,  they  saw  the  benefits  of  current  and 
future  features  such  as  GPS  integration  and  intervisibility,  and  suggested  adding  a  bearing  and 
distance  tool  i.e.  pick  two  points  and  automatically  calculate  bearing  and  distance.  However,  the 
comment  was  made  that  most  of  the  planning  tools  associated  with  the  map  (orders  and  overly 
creation,  intervisibility  etc. )  are  just  for  planning  and  there  is,  “. .  .no  requirement  for  PI  Commander 
to  use  TBCS  for  detailed  movement  planning  and  the  PI  essentially  follows  along  in  combat  team  until 
deployed  for  an  attack.” 

Users  commented  that  grid  reference  display  around  the  map  might  prove  helpful.  This  could  help  to 
convey  the  scale  of  the  “zoom”  of  the  display  as  well  as  help  navigation. 

Additional  information  desired  for  the  map  display  is  map  identification  such  as  sheet  number. 

The  options  for  map  orientation  were  generally  seen  as  useful.  Users  did  not  agree  on  which  option 
was  “the  best”  hence  the  requirement  for  the  options. 

One  user  gave  requirements  for  map  format  from  an  engineering  perspective: 
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•  “Engineer  requires  3  different  maps: 

-  regular  topo  1:50,000 

-  1:50,000  cross  country  movement 

-  1:50,000  road  and  bridge 

Special  zoom  required  for  bridges  to  include  R&B  data 
Special  zoom  for  terrain  to  show  passibility,  sustainability 

Engineer  icons  for  building  material,  heavy  equipment,  local  utilities,  power  lines,  water 
treatment,  etc. 

Feature  to  update  road  maintenance  work,  tactical  bridge  symbols  etc. 

Scale  should  remain  at  1:50,000  to  compare  accurately  to  paper  maps 
Display  points  of  intervisibility” 

The  comments  following  administration  of  the  final  “overall”  questionnaire  reiterated  points  raised 
during  the  trial  i.e.  improved  map  navigation  and  manipulation  is  required. 

3. 6. 4. 2  Map  navigation 

Note  that  map  navigation  refers  to  the  user  moving  the  point  of  interest  around  different  areas  of  the 
map,  not  using  the  map  to  navigate  the  vehicle. 

For  all  three  aspects  of  utility,  ratings  were  consistently  lower  than  for  map  formats. 

Ratings  suggest  that  users  feel  that  map  navigation: 

1.  Was  somewhat  helpful  in  terms  of  allowing 
them  to  operate  effectively  (3.0). 

2.  Was  a  desirable  feature  (3.8). 

3.  Was  somewhat  worse  than  the  current  capability 
(2.7). 

4.  Was  of  moderate  utility  (3.1). 

5.  Usability  in  the  test  bed  was  unacceptable  (2.3). 

6.  Usability  in  the  field  would  be  unacceptable 

(2.0). 

•  Participant  Comments 

Several  issues  were  identified  as  causing  difficulties.  The  major  problem  identified  was  the  inability  to 
smoothly  scroll  the  map  display  across  the  display  window,  since  the  current  functionality  would  only 
allow  the  adjacent  map  regions  to  be  displayed  with  no  common  topography  with  the  previously 
displayed  area.  As  a  result,  users  reported  a  loss  of  situational  awareness  resulting  from  considerable 
difficulty  in  integrating  the  newly  displayed  map  region  with  the  one  they  were  previously  looking  at. 
Users  also  reported  frustration  in  being  unable  to  move  the  map  in  small  increments.  The  predefined 
view  option  was  seen  as  being  useful. 

All  users  had  significant  problems  with  map  navigation.  User  frustration  continued  throughout  the 
trial,  “Map  navigation  -  difficult  to  use”.  Difficulties  were  related  to  time  (refresh),  movement 
distance  and  “scroll”  method.  The  time  problem  caused  frustration  but  most  users  seemed  to 
understand  the  technological  limitation.  More  importantly  users  wanted  a  better  way  of  moving  the 
map  and  suggested  scroll  bars  or  a  drag  feature.  They  thought  a  feature  like  this  would  help  them 
with  the  problem  of  losing  their  situational  awareness  due  to  “jumps”  of  the  map  with  current 
navigation  techniques.  They  said  they  need  to  keep  track  of  location  of  the  current  map  view  in 
relation  to  their  or  other  unit  positions.  The  current  method  of  map  movement  was  not  generally  well 
received.  A  pan,  scroll  or  drag  feature  was  preferred.  The  need  for  simple,  effective  navigation  is 
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very  important.  Many  users  commented  they  need  to  quickly  move  away  from  the  immediate  area  of 
the  combat  team  to  see  either  other  areas  of  interest  or  the  big  picture  (with  more  detail  than  the 
current  overview  provides)  and  quickly  move  back. 

The  map  enlarge  feature  was  seen  as  very  useful  (“. .  .great  piece  of  kit. .  .keep  this  option...”)  by  almost 
all  users,  but  should  be  made  resizable  so  that  it  takes  up  less  space.  However,  they  also  expressed  the 
need  to  see  more  detail  on  this  map  when  required.  One  user  suggested  this  could  be  done  with  full 
size  view  of  the  “enlarge”  view  and  the  zoomed  in  view  by  toggling  between  the  two  as  long  as  both 
maps  indicated  your  own  position. 

3. 6. 4. 3  Map  drawing 

This  feature  was  not  evaluated  because  of  consistent  problems  with  the  system  software. 

3.6.5  Unit  information 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  somewhat  helpful  in  terms  of  allowing 
them  to  operate  effectively  (2.9). 

2.  Was  a  somewhat  desirable  feature  (3.0). 

3.  Was  better  than  the  current  capability  (3.6). 

4.  W as  of  moderate  utility  (3.1). 

5.  Usability  in  the  test  bed  was  clearly  acceptable 
(4.3) 

6.  Usability  in  the  field  would  be  acceptable  (3.6). 


On  question  1 ,  opinion  was  completely  split  on  whether  this  feature  would  allow  users  to  operate 
effectively;  the  2i/c  Infantry  and  Battle  Captain  both  provided  strongly  positive  ratings,  however  the 
Engineering/Recce,  FOO  and  Platoon  Commander  gave  ratings  of  2  or  lower.  On  question  2,  this 
feature  was  seen  to  be  highly  desirable  by  only  the  2i/c  Infantry  and  received  only  moderate  ratings  by 
the  remaining  members  of  the  group. 

•  Participant  Comments 

In  agreement  with  the  ratings  only  the  Coy  2i/c  commented  that  this  information  would  assist  with  his 
daily  collation  of  logistics  information.  All  other  comments  suggested  this  information  would  be 
useful  “at  higher  level  only”,  useful  at  Battle  Group  (BG)  or  Brigade  (Bde)  level,  e.g.  “G3  or  G4”  etc. 
It  appears  that  the  most  useful  information  is  contained  in  “resources”  if  it  can  be  updated  in  near  real 
time.  One  user  commented  that  he  could  not  envision  any  use  of  this  feature. 


1.0  2.0  3.0  4.0  5.0 


Hummsystems 


TBSC/Chameleon  Utility  Trial  Report 


Page  26 


3.6.6  TO&E 

This  feature  comprises  three  components  that  were  evaluated  separately:  Orbat,  resources  and 
information  query  builder. 

3.6.6. 1  Orbat 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  somewhat  helpful  in  terms  of  allowing  them 
to  operate  effectively  (3.1). 

2.  Was  a  somewhat  desirable  feature  (3.1). 

3.  Was  better  than  the  current  capability  (3.7). 

4.  Was  of  moderate  utility  (3.3). 

5.  Usability  in  the  test  bed  was  acceptable  (3.9). 

6.  Usability  in  the  field  would  be  unacceptable  (2.7). 

The  strongest  ratings  for  this  feature  were  provided 
consistently  by  the  Engineering/Recce  for  all  three  questions. 

3. 6.6.2  Resources 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to  operate 
effectively  (3.6). 

2.  Was  a  desirable  feature  (3.6). 

3.  Was  better  than  the  current  capability  (3.6). 

4.  Was  of  moderate  utility  (3.6). 

5.  Usability  in  the  test  bed  was  acceptable  (3.6). 

6.  Usability  in  the  field  would  be  marginally 
acceptable  (3.1). 


3. 6.6. 3  Information  query  builder 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  somewhat  helpful  in  terms  of  allowing  them 
to  operate  effectively  (3.4). 

2.  Was  a  somewhat  desirable  feature  (3.3). 

3.  Was  better  than  the  current  capability  (3.6). 

4.  Was  of  moderate  utility  (3.4). 

5.  Usability  in  the  test  bed  was  acceptable  (3.3). 

6.  Usability  in  the  field  would  be  marginally 
acceptable  (2.9). 

•  Participant  Comments 

Comments  regarding  the  ORBAT  and  Resources  ranged  from  “Excellent  tool  for  easy  access  to  staff 
data  that  may  be  used  in  plan  operations.”,  to,  “ORBATS  not  really  required.”  These  types  of 
comments  came  from  different  senior  people  in  the  trial  combat  team.  Within  this  range  of  comments 
there  was  a  consensus  that  real  information  was  preferred  to  ideal  information  because  it  would  be 
more  useful  in  the  field.  Ideal  staff  planning  information  would  be  useful  for  planning  above  the 
combat  team  level.  Also  the  resource  information  (if  real  and  updated  every  Vi  hour)  was  deemed  as 
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being  very  useful.  Suggestions  of  information  to  be  added  to  enhance  the  resources  were:  daily  battle 
losses,  vehicle  status,  weapon  range,  and  combat  effectiveness. 

The  OC  commented  that  the  information  query  builder  was  “. .  .a  very  good  feature.”  However,  the 
other  team  members  generally  suggested  that  it  would  be  a  useful  tool  for  combat  team  commanders  at 
the  very  lowest. 

Overall  comments  stated  that  in  general  this  feature  is  for  combat  team  command  level  and  planning 
use  only. 

3.6.7  Operation  CEOI 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  major  benefit  in  terms  of  allowing  them  to 
operate  effectively  (4.6). 

2.  Was  a  desirable  feature  (4.3). 

3.  Was  better  than  the  current  capability  (3.9). 

4.  Was  of  strong  utility  (4.2). 

5.  Usability  in  the  test  bed  was  clearly  acceptable 
(4.3). 

6.  Usability  in  the  field  would  be  acceptable  (3.4). 

•  Participant  Comments 

All  comments  were  favourable.  CEOI  was  deemed  to  be  a  very  useful  feature.  Improvements  were  to 
make  it  more  accessible  and  more  secure  than  the  current  implementation.  Also  to  add  more 
information  such  as  :  “Ifeqs,  codeword,  nicknames,  light  recognition  sigs”,  “..Needs  all  information 
represented  in  NATO  orders  format  under  heading  command  and  signals”. 

3.6.8  Status  Bar 

The  next  series  of  items  probed  the  various  status  indicators  within  the  bar  along  the  bottom  of  the 
display  window.  For  the  most  part  responses  to  these  features  were  very  similar,  with  the  exception  of 
the  map  re-centring  mode  and  Vetronics,  the  responses  for  which  are  broken  out  separately. 


— 
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All  of  these  features  were  judged  to  have  strong  utility  and  acceptable  usability,  (see  composite  figures 
below). 


1.0  2.0  3.0  4.0  5.0 


Figure  3A:  Composite  ratings  Message  Status 


Figure  3B:  Composite  ratings 
Cursor  Status 
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3.6.8. 1  Message  Status 

•  Participant  Comments 

The  message  status  bar  is  quite  small  and  users  commented  that  it  is  difficult  to  see,  especially  when  in 
a  vehicle.  An  indication  is  required  to  alert  users  to  “immediate”  messages.  Also  a  better  indication 
of  priority  messages  is  needed. 

3. 6.8.2  Cursor  Position  Indicator 

•  Participant  Comments 

Comments  were  related  to  size  and  the  fact  that  it  is  difficult  to  see.  This  would  be  relied  on  less  if  a 
grid  indication  were  provided  around  the  map. 

3. 6. 8. 3  Date  and  Time  Indicator 

One  user  commented  that  this  was  too  small.  Another  commented  that  the  clocks  must  be 
synchronised  to  the  nearest  second  with  all  other  units  otherwise  it  should  not  be  displayed.  Another 
users  commented  that  there  were  too  many  options. 

3. 6. 8. 4  Active  Unit  Position 

Users  generally  commented  that  having  the  grid  references  instantly  available  was  a  desirable  feature. 

3.6.8. 5  Cursor  Status 

Cursor  position  indicator  choices  were  seen  as  being  helpful  to  combined  options. 


3.6.8.6  Map  Re-centring  Mode 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to  operate 
effectively  (4.0). 

2.  Was  a  desirable  feature  (4.0). 

3.  Was  somewhat  better  than  the  current  capability 
(3.4). 

4.  Was  of  moderate  utility  (3.8). 

5.  Usability  in  the  test  bed  was  acceptable  (3.3). 

6.  Usability  in  the  field  would  be  marginally 
acceptable  (3.1). 


•  Participant  Comments 

Re-centre  was  helpful  to  users  once  they  understood  the  various  options.  Users  commented  that  re¬ 
centre  on  own  position  should  not  occur  on  every  move  as  it  takes  too  much  time  to  update. 


Users  commented  that  they  did  not  want  to  rely  on  map  re-centring  alone  and  would  prefer  “. .  .more 
flexible  map  browsing”,  (this  relates  to  previous  comments  on  map  scrolling  or  panning.) 
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3.6.8.7  Vetronics 

Ratings  suggest  that  users  feel  that  this  feature: 

1.  Was  helpful  in  terms  of  allowing  them  to 
operate  effectively  (4.3). 

2.  Was  a  desirable  feature  (3.7). 

3.  Was  better  than  the  current  capability  (3.8). 

4.  Was  of  moderate  utility  (3.9). 

5.  Usability  in  the  test  bed  was  clearly  acceptable 
(4.2). 

6.  Usability  in  the  field  would  be  acceptable  (3.8). 

For  question  2,  only  the  Combat  Team  Commander  saw  this  as  being  worse  than  the  current 
capability.  In  general,  participants’  ratings  for  this  feature  tended  to  be  higher  than  might  be  expected 
from  the  comments  indicated  below. 

•  Participant  Comments 

Users  viewed  this  feature  with  some  scepticism  i.e.  they  didn’t  believe  it  would  be  implemented  in  the 
CF.  Flowever,  some  users  mentioned  that  a  warning  of  an  impending  vehicle  system  failure  would  be 
helpful  especially  if  integrated  with  all  vehicle  systems  for  units  and  higher  (i.e.  a  logistics  function). 
Other  users  said  they  had  a  driver  to  take  care  of  the  vehicle  and  did  not  need  the  feature. 

3.6.9  System  Options 

Participants  were  required  to  use  individual  segments  of  “system  options”  features  i.e.  “messaging” 
and  “viewer”.  Each  of  these  features  was  not  systematically  evaluated  during  the  trial.  In  general, 
participants  were  surprised  by  the  amount  of  custom  configuration  available  to  them  and  although  they 
did  not  have  lengthy  experience  reviewing  their  effects,  they  found  the  utility  very  promising. 

3.7  Post  Scenario  Focus  Group  Discussion 

3.7.1  Overall  Ratings  of  Major  System  Features 

Prior  to  the  general  focus  group  discussion,  participants  provided  an  overall  questionnaire  rating  for  six 
of  the  most  frequently  used  individual  features  in  terms  of  utility  and  ease  of  use.  Mean  data  are  shown 
below  and  give  a  general  impression  of  participant  perception  following  the  trial. 
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Table  3:  Mean  Overall  Ratings  for  Selected  Features  across  all  participants. 


Feature 

Utility 

Ease  of  Use 

Overlays 

4.1 

2.9 

Messaging 

4.2 

3.2 

Symbols 

4.2 

3.4 

Map 

3.1 

2.9 

Unit  Info. 

4.2 

4.0 

TO&E 

2.9 

3.9 

Mean 

3.8 

3.4 

While  the  overlay,  messaging,  symbols  and  unit  information  are  generally  seen  as  having  strong 
utility,  the  map  and  TO&E  features  were  rated  as  having  moderate  to  less  than  acceptable  utility.  Only 
unit  information  and  TO&E  were  clearly  seen  as  having  acceptable  ease  of  use. 

It  should  be  noted  that  the  combat  team  commander  had  consistently  higher  ratings  than  those  of  other 
combat  team  members  and  therefore  distorted  somewhat  the  mean  values.  Without  this  data  the  mean 
utility  and  ease  of  use  values  drop  to  3.2  and  2.7  respectively.  These  data  suggest  that  neither  the 
overall  utility  nor  ease  of  use  of  the  principal  system  features  were  viewed  with  much  favour  by  most 
of  the  combat  team. 

3.7.2  Review  of  Major  TBCS/Chameleon  Issues 

Following  the  last  stage  of  the  scenario,  a  general  focus  group  discussion  was  conducted  to  review  the 
high  utility  features  of  TBCS/Chameleon  (based  on  the  trial  administrators’  observations  and 
participants’  comments),  as  well  as  users’  initial  perception  of  future  TBCS/Chameleon  features. 

After  2/i  days  exposure  to  the  software,  users  made  the  following  positive  comments  about  system 
features/functions. 

•  The  map,  overlays,  symbols  and  TO&E  (including  the  playback  of  moves)  features 
combine  to  form  effective  planning  and  training  tool. 

•  The  video  clip  option  will  provide  useful  recce  information  (users  suggest  this  should  be 
resizable). 

•  CEOl  decreases  workload  in  gathering  and  recalling  this  information  ( users  assumed  this 
would  be  filled  with  all  the  relevant  information  they  have  access  to  now). 

•  GPS  integration  with  map  will  increase  situation  awareness. 

•  The  Automation  capability  provided  by  messaging  e.g.  logistics,  MASH,  Cas  evac.  will 
reduce  workload. 

•  Alarm  on  approach  of  mine  field  will  increase  safety  and  situation  awareness. 

•  Messaging  allows  for  an  increase  in  accuracy  (everyone  receives  the  same  information 
with  no  transcription  errors). 

•  Message  pre-formatting  reduces  the  workload  associated  with  preparing  messages. 

3.7.3  Future  Issues 

Major  future  issues  included  in  the  discussion  were  interface  implementation  and  decision  aids.  The 
concepts  of  integrating  features  such  as  touch  screen,  stylus  and  voice  recognition  interaction  were 
generally  well  received.  Participants  were  generally  familiar  with  these  features  in  some  form  or 
another  (e.g.  hank  machines)  and  saw  high  utility  in  these  modes  of  interface.  However,  the  concept 
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of  a  head  up  display  was  not  well  understood  due  to  lack  of  familiarity.  No  comment  can  be  made  on 
the  potential  utility  of  this  feature. 

Decision  aids  were  mentioned  as  having  high  utility  throughout  the  trial,  and  this  theme  was  picked  up 
during  the  final  focus  group  discussions.  In  particular  decision  aids  were  seen  to  be  most  useful  in  the 
context  of  planning  activities  and  include: 

•  weapon  range  indicator, 

•  bearing  and  range  tool, 

•  integration  of  GPS  and  laser  range  finder, 

•  intervisibiliy  tool,  and 

•  route  planning  tools. 

3.8  Rating  differences  across  positions 

With  only  one  individual  to  provide  a  rating  for  each  position,  any  interpretation  of  rating  data  for  a 
position  must  be  treated  with  caution.  The  data  could  reflect  either  true  differences  resulting  from 
different  needs  and  perceptions  across  positions,  or  could  just  be  the  result  of  differences  among 
individuals.  Notwithstanding  this  limitation,  mean  data  for  all  three  utility  questions  and  both 
usability  questions  have  been  compiled  over  all  system  features  and  are  shown  in  Table  4  below. 

These  data  show  that  the  utility  of  the  current  TBCS  feature  set  is  seen  as  moderately  positive  across 
the  entire  combat  team. 


Table  4:  Overall  Utility  and  Usability  Ratings  for  each  combat  team  position 


Mean  Utility  Rating 

Mean  Usability  Rating 

Combat  Team  Commander 

3.9 

4.5 

Battle  Captain 

3.8 

3.4 

Troop  Leader 

3.7 

2.9 

2i/c  Infantry 

3.8 

3.5 

Platoon  Commander 

3.7 

3.6 

Engineering  Recce 

3.7 

2.4 

Artillery  FOO 

3.6 

3.2 

With  respect  to  usability  issues,  there  is  much  less  agreement  across  the  team.  The  Troop  Leader  and 
Engineering  positions,  on  average,  rate  the  system  ease-of-use  as  unacceptable.  At  the  other  extreme, 
the  Combat  Team  Commander  rated  all  features  on  average  as  being  easy  to  use.  The  remaining  team 
members  fell  somewhere  between  these  extremes. 

3.9  Desirability  of  system  features 

The  following  Table  provides  a  further  breakdown  of  the  desirability  of  system  features  (question  2 
Annex  C  -  Utility  Trial  Questionnaire)  across  combat  team  positions,  and  contains  only  those  features 
which  received  a  rating  of  four  or  higher  (shaded  cells).  This  information  may  be  of  some  assistance 
in  future  development  of  the  system  when  trade-offs  have  to  be  made  between  system  features  (and 
their  overall  utility  across  the  team)  against  other  constraints.  These  data  may  also  be  of  use  if  there  is 
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some  consideration  given  to  matching  the  specific  system  functionality  to  the  different  needs  of 
different  combat  team  positions. 


Table  5:  Features  rated  as  being  moderately  desirable  or  higher  for  a  future  system. 


Combat 
Team  Comd 

Battle 

Captain 

2i/c  Infantry 

Platoon 

Comd 

Troop 

Leader 

Eng  Recce 

Artillery 

FOO 

Overlay:  Control 
measures 

Not  Rated 

Overlay:  Co-ord 
plans 

Overlay:  Orders 

Mess:  General 

Mess:  Contact  Rep 

Mess:  Wng  O 

Mess:  Send  overlay 

Symbols:  Frunit 

Not  Rated 

Symbols:  En  unit 
editor 

Map:  Fonnat 

Map:  Navigation 

Unit  information 

TO&E:  Orbat 

TO&E:  Resources 

TO&E:  Info  query 

Op:  CEOI 

Mess.  Status 

Cursor  position 

Date  time  indicator 

Active  unit  position 

Cursor  mode 

Map  recentre  mode 

Vetronics 

Not  Rated 

3.10  Summary 

In  this  summary,  the  composite  data,  collapsed  over  all  combat  team  positions,  will  be  considered 
from  a  variety  of  perspectives  in  order  to  better  understand  the  overall  trends.  Since  the  primary 
purpose  of  the  trial  was  on  the  utility  of  TBCS  as  a  requirements  capture  tool,  the  rating  data  on  the 
three  utility  questions  will  be  the  major  area  of  focus. 
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Table  6  Utility  of  System  Features 
(mean  of  Questions  1,  2,  3) 

In  terms  of  gross  measures  of  utility,  taken  across  all 
system  features,  the  mean  rating  on  question  1  of  Annex 
C  -  Utility  Trial  Questionnaire,  was  3.8,  which  suggests 
that  participants  overall  judged  that  the  system  features 
would  provide  a  moderately  positive  benefit  to  their 
operational  effectiveness.  For  question  2,  the  mean 
rating  was  4.0  which  would  indicate  that  across  all 
features  were  seen  to  be  moderately  desirable  in  a  future 
system.  For  question  3,  the  mean  response  of  3.5 
suggests  that  the  system  features  were  not  seen  to  be 
much  of  an  improvement  over  present  capability.  This 
judgement  may  have  been  influenced  by  usability 
concerns,  since  ratings  on  the  ease  of  use  questions  were 
overall  lower  for  most  system  features  than  for  the 
utility  questions 

To  further  summarise  the  utility  data,  an  overall  rating 
of  utility  for  each  system  feature  has  been  calculated 
across  all  three  questions  and  all  trial  participants. 

These  data  are  shown  in  table  6,  with  the  system 
features  ranked  in  descending  order  of  utility. 

It  is  interesting  to  note  that  many  of  the  elements  which 
appear  in  the  TBCS  status  bar  are  ranked  towards  the 
top  of  the  list.  It  should  be  noted  that  for  the  most  part, 
these  features  (which  provide  a  convenient  summary  of 
status  information)  contribute  more  to  general 
situational  awareness  than  they  do  to  the  actual 
execution  of  specific  tasks.  Further,  these  features  also 
received  moderate  to  moderately  high  ratings  on 
usability,  and  there  is  a  strong  possibility  that  the  trial  participants  were  unable  to  set  aside  their 
impressions  of  ease-of-use  when  making  judgements  about  the  utility  of  a  feature.  Notwithstanding 
this  possibility,  it  should  be  noted  that  no  system  feature  was  rated  below  3  in  terms  of  average  utility, 
suggesting  that  all  features  were  seen  to  be  useful  for  including  in  the  final  system 

One  further,  useful  method  of  summarising  all  of  the  data,  is  to  organise  the  system  features  into  four 
broad  categories  of  utility:  moderate  (mean  rating  3-3.9)  or  high  (mean  rating  4+)  and  acceptable 
usability  (mean  rating  >3)  or  unacceptable  (mean  rating  <3).  For  the  utility  categorization,  only  the 
responses  to  the  question  on  ease-of-use  in  the  field  were  considered  (question  5),  since  these  provide 
more  meaningful  feedback  for  system  development  purposes  than  the  ratings  on  ease-of-use  in  the  test 
environment 


System  Feature 

Mean  Rating 

Cursor  Position 

4.9 

Message  Status 

4.7 

Date/time  indicator 

4.7 

Active  Unit  position 

4.7 

Message:  Wng  0 

4.6 

Map:Format 

4.4 

Op:  CEOI 

4.3 

Cursor  Status 

4.3 

Messaging:  General 

4.3 

Messaging:  Send 
overlay 

4.3 

Messaging:  ConRep 

4.1 

Map  re-centre 

4.0 

Map:  Navigation 

3.9 

Symbols:  Fr  unit 

3.7 

Vetronics 

3.7 

Symbols:  En  unit  ed 

3.6 

TO&E:  Resources 

3.5 

Overlay  :Con 

3.5 

measures 

Overlay:  Coord  plans 

3.4 

TO&E:  Info  query 

3.2 

Overlay:  Orders 

3.2 

TO&E:  Orbat 

3.1 

Unit  info 

3.0 
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The  results  of  this  classification  are  shown  in  Table  7  below. 


Table  7:  System  Features  classified  by  utility  and  ease  of  use. 


High  Utility 

Moderate  Utility 

Acceptable  Usability 

Messaging:  general 

Messaging:  warning  order 

Operation:  CEOI 

Message  status  indicator 

Date/time  indicator 

Active  unit  position 

Map  recentre  mode 

Vetronics 

Symbols:  friendly  units 

Unit  information 

TO&E:  Resources 

Vetronics 

Map  drawing 

Unacceptable  Usability 

Messaging:  contact  report 

Map  format 

Map  navigation 

Overlays:  control  measures 

Overlays:  co-ordinate  plans 

Overlays:  orders 

Symbols:  enemy  units  editor 

TO&E:  Orbat 

TO&E:  Information  query  builder 

This  table  provides  useful  directions  for  guiding  short  term  system  development  priorities  if  future 
user  trials  are  contemplated.  For  example,  features  which  are  shown  to  have  high  or  moderate  utility 
and  low  usability  could  be  carefully  reviewed  with  the  intention  of  developing  a  much  improved 
interface  and/or  functionality. 
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4 


Discussion 


This  section  draws  together  some  of  the  major  issues  which  emerged  from  the  utility  trial.  The  topics 
to  be  covered  are: 

•  trial  participants 

•  map  usage 

•  communication  issues 

•  access  to  resource  information 

•  TBCS/Chameleon 

•  utility  versus  ease-of-use. 

Where  possible,  utility  related  issues  are  followed  by  design  recommendations.  Please  note  that 
recommendations  have  been  numbered  consecutively  across  all  topics  for  ease  of  reference. 

4.1  Trial  participants 

At  the  start  of  the  Results  section  we  reviewed  the  characteristics  of  the  participants  for  the  utility  trial 
and  showed  how  they  fell  short  in  many  ways,  compared  with  the  desired  sample  requested.  We  now 
outline  some  concerns  about  the  test  participants  based  upon  a  general  subjective  evaluation  from 
watching  their  performance  in  the  more  structured  aspects  of  the  trial,  and  from  interpreting  their 
comments  in  the  trial  discussion  sessions.  While  such  a  subjective  evaluation  has  potentially  low 
reliability,  the  two  test  administrators  have  significant  field  experience  in  working  with  military 
personnel  in  a  wide  range  of  trial  environments.  Hence,  we  have  a  reasonable  knowledge  base  against 
which  we  can  compare  the  present  trial  participants. 

We  believe  that  there  are  five  main  issues  relating  to  the  trial  participants  that  give  rise  to  concerns. 
These  are: 

1 .  The  degree  to  which  they  are  a  representative  sample, 

2.  Their  ability  to  play  the  role  in  question, 

3.  Their  ability  to  play  two  different  roles  during  the  trial, 

4.  Their  ability  to  use  their  imagination  to  evaluate  the  role  the  system  could  play  in  other 
contexts,  and 

5.  Their  ability  to  imagine  how  additional  functionality  might  be  suitable  for  certain 
operational  tasks. 

We  now  consider  each  of  these  points  in  turn. 

Given  the  limited  experience  of  many  of  the  participants,  particularly  in  the  roles  which  they  were 
scheduled  to  play,  we  have  strong  reservations  concerning  the  degree  to  which  the  participants  actually 
are  representative  of  typical  combat  teams.  Our  conclusion  is  that  their  knowledge  base  was  shallow 
and  in  many  cases  they  had  limited  experiences  in  performing  a  range  of  tasks  under  operational  or 
exercise  conditions.  While  this  lack  of  experience  is  unlikely  to  bias  the  evaluations  in  any  systematic 
way,  it  does  suggest  that  caution  should  be  applied  in  considering  the  results  to  be  representative  of  the 
army  population  who  are  likely  future  users  of  TBCS/Chameleon.  That  is,  unless  the  actual  army 
population  itself  is  equally  low  in  experience.  If  this  is  the  case,  then  it  makes  this  and  future 
evaluations  of  the  system  more  problematic,  since  potential  users  may  have  so  little  knowledge  about 
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the  tasks  done  under  operational  conditions  that  evaluating  the  system  without  this  contextual 
experience  represents  a  major  risk  for  identification  of  the  appropriate  suite  of  functions  required. 

The  lack  of  experience  among  the  trial  participants  also  impacted  upon  the  test  participants’  ability  to 
play  the  role  expected  in  the  execution  of  the  scenario.  They  frequently  needed  to  be  prompted  to  go 
beyond  the  basic  tasks  that  formed  the  core  of  the  scenario  segments.  In  some  cases,  other  members  of 
the  team,  whose  own  task  performance  was  dependent  on  tasks  perform  by  others,  prompted  the  team 
member  in  question  to  do  the  task  that  was  missed.  The  result  is  that  the  software  did  not  get  the  full 
shakeout  that  one  might  have  hoped  for. 

In  retrospect,  it  appears  that  the  idea  of  having  some  participants  play  two  different  roles  (in  order  to 
economise  on  the  number  of  users  required)  was  not  successful.  We  found  that  we  had  to  constantly 
remind  those  playing  two  roles  to  switch  to  the  alternate  role  and  carry  out  the  mission-appropriate 
tasks.  This  conclusion  is  reinforced  by  the  fact  that  for  those  mission  segments  where  the  participant 
had  to  assume  two  roles,  no  separate  post-segment  ratings  for  each  of  the  individual  roles  were 
obtained. 

Although  one  or  two  of  the  trial  participants  showed  a  lot  of  imagination,  this  was  not  true  of  the 
majority.  This  expressed  itself  in  two  ways.  First,  they  could  not  go  beyond  the  immediate  limited 
range  of  tasks  and  existing  procedures  to  speculate  on  how  the  system  might  support  other  tasks,  or 
how  other  tasks  could  be  potentially  served  by  a  BMS.  Second,  if  some  aspect  of  the  system  interface 
or  functionality  was  giving  them  problems  in  performing  a  specific  task,  they  seemed  unable  to  set 
aside  this  aspect  in  evaluating  how  the  system  might  effectively  support  other  tasks  and  activities. 

In  conclusion,  these  limitations  of  the  trial  participants  probably  result  in  an  underestimation  of  both 
the  potential  utility  and  general  applicability  of  TBCS/Chameleon,  since  they  have  had  sufficient 
experience  to  extrapolate  to  mission  contexts  and  where  the  system  may  be  able  to  play  a  significant 
role. 

4.2  Networked  system  stability 

As  discussed  in  the  Results  section,  the  networked  TBCS/Chameleon  software  system  crashed  more 
frequently  than  expected.  The  software  designers  worked  very  hard  throughout  the  trial  to  increase  the 
stability  of  the  system  by  creating  and  installing  software  patches  half  way  through  the  utility  trial. 

This  increased  the  stability  somewhat  but  still  resulted  in  some  residual  instability. 

This  instability  of  the  networked  TBCS/Chameleon  system  affected  the  users  ability  to  objectively 
assess  the  utility  of  the  software.  It  was  anticipated  that  many  of  the  features  would  be  non  functional 
and  that  there  would  be  some  “glitches”  in  the  software.  However,  considerable  time  was  used  in 
rebooting  the  network  and  users  became  frustrated  by  the  constant  interruptions  that  resulted.  While 
users  were  generally  able  to  separate  utility  issues  from  ease  of  use  there  is  a  potential  for  considerable 
system  instability  to  affect  system  utility  ratings.  Therefore,  it  is  likely  that  there  was  a  slight 
underestimation  of  system  utility  as  a  result. 

4.3  Map  issues 

Given  the  centrality  of  map  use  for  many  combat  team  activities,  it  is  not  surprising  that  this  area  of 
system  functionality  generated  some  of  the  most  extensive  comments  and  discussions.  Three  major 
areas  relating  to  map  use  emerged:  size,  navigation  and  content. 
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4.3.1  Map  size 

At  the  most  general  level  the  greatest  concern  was  the  small  size  of  the  viewable  area.  This  was  the 
particular  concern  for  the  senior  combat  team  personnel  who  believed  that  the  format  reduced  their 
ability  to  integrate  a  more  local  view  of  the  area  of  interest  with  the  more  global  picture.  It  appeared 
that  the  capability  for  providing  a  scalable  window  insert  to  show  other  areas  of  the  map,  was  not  an 
effective  solution  for  providing  the  necessary  integration  of  situation  awareness.  These  users  believed 
that  they  would  still  need  a  paper  map  to  provide  the  more  global  view.  Some  users  also  expressed 
concern  over  the  total  size  of  the  map  in  that  it  did  not  match  the  size  that  they  were  used  to  having  for 
combat  team  operations. 

A  related  issue  to  map  size,  is  the  immediate  loss  of  part  of  the  map  visible  area  that  occurs  when  a 
message  is  received  and  occludes  part  of  the  underlying  map  area.  Users  suffer  a  major  loss,  or 
disruption,  in  situation  awareness  when  this  happens.  Additional  workload  is  created  and  there  is  a 
loss  of  effectiveness,  as  the  user  has  to  restore  situation  awareness  once  the  message  has  been  removed 
from  the  screen. 

4.3.2  Map  navigation 

This  area  produced  significant  problems  and  comments  from  all  users.  While  we  discount  some  of  the 
difficulties  users  encountered,  which  we  ascribe  to  limitations  in  the  current  speed  of  the  system,  there 
is  no  doubt  that  the  ability  to  navigate  smoothly  and  quickly  around  the  map  is  a  major  ease-of-use 
issue.  Because  of  the  difficulties  users  encountered  in  map  navigation,  the  perceived  utility  of  the  map 
was  probably  underestimated.  Smooth  map  navigation  is  at  the  heart  of  the  user's  ability  to  maintain 
situation  awareness.  That  is,  to  provide  the  appropriate  mental  picture  users  need  to  be  able  to  quickly 
integrate  information  within  the  existing  map  window  with  information  from  either  immediately 
adjacent  areas,  or  areas  that  are  more  distal.  Time  is  of  the  essence  in  achieving  this  integration,  as  the 
user  must  mentally  maintain  one  spatial  perspective  while  supplementing  or  integrating  with  it  other 
spatial  information.  Users  provide  several  examples  of  how  the  current  system  fails  to  support  the 
various  types  of  map  navigation  activity  that  are  needed  to  support  and  enhance  situation  awareness; 
these  were  outlined  in  more  detail  in  the  results  section. 

4.3.3  Map  content  and  drawing  features 

In  general,  users  were  satisfied  that  TBCS/Chameleon  was  able  to  show  existing  map  formats  in  more 
or  less  the  level  of  detail  and  rendering  that  is  found  in  the  existing  paper  map  format.  It  was  clear  that 
users  saw  a  significant  potential  for  improving  map  functionality  with  the  advent  of  computer  based 
map  technology.  The  major  theme  to  emerge  was  the  need  for  differential  map  detail  for  different 
mission  contexts,  different  mission  tasks,  different  time  pressures  and  different  team  roles.  For 
example,  the  levels  of  detail  to  support  planning  before  an  advance,  while  on  the  move,  conducting  a 
hasty  attack,  and  consolidation  were  all  quite  different.  Users  also  expressed  a  strong  desire  to  see 
certain  types  of  information  available  on  a  map  that  could  not  be  made  currently  available  in  existing 
formats.  The  specific  information  required  varied  across  combat  team  position,  for  example,  infantry 
wanted  to  see  more  detail  concerning  the  terrain  and  buildings,  whereas  engineering  was  more 
concerned  over  issues  relating  to  bridges,  roads  and  utilities. 

The  display  of  friendly  symbols  on  the  map  seemed  to  be  useful,  however  the  associated  information 
in  the  pop-up  box  was  seen  to  be  more  useful  for  logistics  activities  than  for  combat  operations.  Users 
were  positive  about  several  aspects  of  the  functionality  that  enhanced  their  situation  awareness.  These 
included  the  option  to  filter  units  and  show  aggregations  and  the  type  of  unit  information  displayed 
(e.g.  weapons,  Tac  or  call  sign).  Users  made  several  suggestions  for  enhancements  to  the  utility  and 
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implementation  of  this  feature  to  further  enhance  its  overall  effectiveness  for  use  in  the  field.  In 
contrast,  the  enemy  units  editor  was  not  seen  to  be  useful  or  usable  for  combat  operations. 

The  general  conclusion  that  must  be  drawn  is  that  to  fully  support  all  combat  team  members  and  all 
aspects  or  combat  team  operations,  a  flexible  map  format  is  required.  This  would  allow  the  user  to 
select  the  appropriate  level  of  detail  symbology  and  tools  for  the  immediate  mission  task  at  hand. 

4.3.4  Map  tools 

Users  were  generally  enthusiastic  about  the  potential  implementation  of  several  map  tools  and  features 
that  might  be  expected  in  a  future  evolution  of  the  TBC  S/Chameleon  concept.  These  included 
intervisibility,  calculation  of  bearing  and  distance  between  points  and  GPS  integration  with  a  laser 
range  finder. 

Overall,  the  conclusion  that  must  be  drawn  from  the  utility  trial  is  that  users  place  a  high  priority  on 
map  usage  to  create  and  maintain  both  local  and  global  situation  awareness.  They  are  used  to  a  paper 
map  format,  which  they  have  learned  to  use  effectively  to  integrate  these  two  aspects  of  situation 
awareness.  While  they  see  the  electronic  map  format  as  having  strong  potential  to  improve  situational 
awareness  over  the  paper  map,  many  aspects  of  the  TBCS/Chameleon  implementation  of  map 
functionality  produce  a  lower-level  of  situation  awareness  than  exists  with  their  present  interaction 
with  the  paper  format.  Without  major  changes  in  features  to  support  the  required  utility,  and  a  parallel 
improvement  in  the  interface  to  support  usability,  the  likely  result  of  fielding  a  system  with  the  current 
capability  is  that  users  will  probably  continue  to  use  a  traditional  map  format  in  conjunction  with 
TBCS  Chameleon,  with  no  guaranty  of  any  improvement  to  their  situation  awareness.  It  is  also 
possible  that  such  a  hybrid  approach  could  actually  impair  situation  awareness,  increase  workload  and 
decreased  combat  team  effectiveness. 

4.3.5  Recommendations 

1)  Increase  the  size  of  the  viewing  area  of  the  map  display. 

2)  Reduce  the  amount  of  map  area  that  is  obscured  by  interface  elements. 

3)  Provide  appropriate  map  formats  and  drawing  tools  which  are  better  matched  to  the  requirements 
of  combat  team  members  (e.g.  engineering  requires  different  map  content  from  artillery). 

4)  Allow  users  to  more  easily  integrate  map/trace  information  and  message  text  information.  Users 
need  to  simultaneously  view  map  features  and  message  information. 

5)  Modify  map  “pan”  feature  to  allow  smooth  movement  of  the  map  without  the  current  “jumps” 

(e.g.  allow  the  user  the  capability  to  drag  or  scroll  the  map  as  little  as  %”  in  any  direction,  or  as 
much  as  one  complete  screen  width  while  allowing  the  user  to  see  the  map  throughout  the 
movement). 

6)  Allow  quick  method  of  alternating  between  two  map  views  (i.e.  in  both  location  and  zoom  factor). 

7)  Provide  the  enlarge  view  feature  with  a  resizable  window  capability. 

8)  Provide  a  capability  to  select  an  area  within  the  enlarge  view  window  to  become  the  current  map 
display  view. 

9)  Provide  a  tool  that  allows  users  to  select  two  points  and  automatically  calculate  bearing  and 
distance. 
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10)  Allow  users  the  option  to  display  major  grid  reference  marks  (i.e.  eastings  and  northings)  at  the 
edge  of  the  map  without  compromising  recommendation  2. 

11)  Provide  more  flexible  drawing  tools  to  allow  rapid  annotation  of  drawings  such  as  quick  sketches 
for  a  hasty  attack. 

12)  Provide  a  readily  accessible  count  down  clock  feature  with  user  selectable  alarm  increments  (e.g.  a 
warning  5  minutes  before  time  equals  H-hr). 

4.4  Communication  issues 

Like  map  usage,  communication  is  a  major  component  of  combat  team  operations.  Users  place  great 
stress  on  having  fast  and  accurate  communication  achieved  with  the  minimum  of  effort  and  mental 
workload.  The  specific  topics  we  will  comment  upon  include  general  utility,  contact  reports,  orders 
and  sending  overlays. 

4.4.1  General  utility 

While  users  generally  saw  significant  potential  for  this  form  of  communication,  simply  executed  with 
a  few  quick  and  over-learned  spoken  words,  a  major  area  of  concern  expressed  was  the  time  it  takes  to 
create  many  basic  messages.  Users  wanted  to  see  the  message  creation  interface  made  much  simpler 
and  wanted  to  use  pre-formatted  generic  content  as  much  as  possible. 

A  second  issue  of  concern  was  message  reception,  where  two  problem  areas  emerged.  First,  users  did 
not  believe  that  they  were  receiving  sufficient  situation  awareness  of  new  messages,  in  that  an 
increment  in  the  number  displayed  in  the  message  in-tray  status  bar  could  be  easily  missed  if  their 
attention  were  elsewhere.  In  particular,  users  felt  that  they  should  be  alerted  in  particular  to  high 
priority  incoming  messages,  and  the  way  this  was  currently  implemented  failed  to  meet  this  need. 
Secondly,  the  accessed  messages  covered  too  much  of  the  underlying  map  area  with  the  result  that  it 
was  often  difficult  to  do  the  necessary  integration  between  the  message  content  and  what  was 
happening  on  the  map. 


4.4.2  Recommendations 

13)  Change  the  “three  tab”  message  format  to  one  window. 

14)  Allow  the  user  to  define  a  default  recipient  list  for  all  messages. 

15)  Allow  messages  to  be  sent  with  incomplete  information.  For  example  a  contact  report  could  be 
sent  by  making  the  following  clicks:  message  -  contact  -  send  (results  in  an  automatic  contact 
wait  out)  or  message  -  contact  -  location  on  map  -  send  (results  in  contact  wait  out  with  grid 
reference  information). 

16)  Provide  better  means  of  rapid  situation  awareness  of  reception  of  priority  message  through  visual 
cues  (while  maintaining  the  auditory  cue  option).  The  default  mode  for  reception  of  priority 
messages  should  be  set  to  “on”. 

17)  Improve  the  editing  capability  of  messaging  to  support  order  production  to  reduce  the  number  of 
steps  involved  in  numerous  cut  and  paste  operations. 

1 8)  Provide  the  capability  for  message  forwarding. 
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19)  Continue  to  develop  preformatted  messages  matched  to  the  needs  of  high  frequency  tasks 
performed  by  different  team  members. 

20)  For  messages  that  rely  on  a  common  time  base  (e.g.  “orders  in  x  minutes”),  provide  a  reference 
time  stamp  integrated  with  preformatted  messages. 

21)  Provide  a  capability  to  allow  the  sender  of  the  certain  message  to  check  an  easily  referenced  list, 
which  shows  who  received,  reviewed  and  understood  messages. 

4.4.3  Contact  reports 

The  analysis  of  contact  report  issues  can  be  centred  on  two  areas,  sending  and  receiving.  Users 
expressed  considerable  concern  over  the  time  it  would  take  in  TBCS/Chameleon  to  send  a  contact 
report,  compared  with  how  quickly  this  can  be  accomplished  with  voice  communication.  In  general, 
users  would  prefer  fewer  steps,  fewer  options  and  more  pre-formatted,  easily  accessible  forms,  for 
example,  having  available  a  quick  pick  list  of  expected  enemy  units.  However,  users  also  saw 
considerable  potential  for  quickly  sending  certain  types  of  information  if  data  derived  from  laser  range 
finding  could  be  integrated  into  the  TBCS/Chameleon  system. 

In  terms  of  message  reception,  users  saw  great  potential  for  having  some  of  the  contact  report 
information  appear  directly  on  the  map,  as  currently  implemented  in  TBCS/Chameleon.  They 
typically  saw  this  feature  as  reducing  workload  and  communication  errors,  enhancing  situation 
awareness  for  new  information  and  integrating  that  information  into  their  mental  picture.  The 
enhancements  to  situation  awareness  include:  immediate  comprehension  of  the  nature  of  the  contact, 
the  exact  location  and  its  movement.  There  were  two  aspects  relating  to  the  reception  of  contact 
reports  that  produced  some  concerns  and  negative  comments.  First,  the  user  is  totally  unaware  of 
contact  reports  that  appear  outside  of  the  map  area  that  is  currently  displayed,  even  though  they  may 
be  of  critical  importance.  Second,  even  within  the  map  area  displayed,  a  new  contact  report  may  fail 
to  attract  the  necessary  attention  and  can  be  easily  overlooked.  This  becomes  more  critical  as  the 
amount  of  information  displayed  on  the  map  increases  and  as  the  user  narrows  attention  to  focus 
intently  on  a  task. 

Users  believed  that  the  symbol  information  plotted  on  the  map  should  be  as  simple  as  possible  and 
allow  them  to  quickly  interrogate  additional  information  with  a  simple  button  click.  There  is  a  further 
need  to  support  situation  awareness  by  reducing  display  clutter  arising  from  multiple  contacts  as  a 
mission  unfolds  by  allowing  changes  in  unit  status  to  be  reflected  in  the  displayed  symbology,  for 
example  in  the  case  of  contacts  which  have  been  destroyed. 

The  final  central  area  of  major  concern  that  relates  to  both  message  transmission  and  reception  is  the 
perceived  time  it  would  take  for  a  "contact  wait  out".  Some  users  believed  that  it  was  necessary  to 
continue  to  circulate  this  information  by  voice  because  of  the  ease  in  which  it  can  be  accomplished, 
the  potential  expression  of  urgency  by  voice  intonation  and  the  alerting  effect  this  has  upon  the 
recipient,  particularly  if  attention  was  concentrated  elsewhere.  On  the  other  hand  other  users  believed 
that  some  of  these  needs  could  be  addressed  in  a  well-designed,  visual-spatial  format,  which  would 
also  have  the  advantage  of  providing  immediate  situation  awareness  of  the  contact  location. 
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4.4.4  Recommendations: 

22)  Provide  the  capability  to  flag  the  user  that  off  screen  contact  reports  have  occurred. 

23)  Visually  alert  the  user  to  a  new  contact  reports  plotted  on  the  map. 

24)  Provide  the  capability  to  alter  the  appearance/status  of  enemy  symbols  (e.g.  to  show  enemy 
destroyed). 

25)  Provide  the  capability  to  quickly  interrogate  by  way  of  clicking  an  enemy  unit  symbol  for 
information  in  the  original  contact  report  (e.g.  who  made  the  report,  at  what  time,  what  was  the 
enemy  doing  etc). 

4.4.5  Orders 

The  perceived  utility  of  the  TBCS/Chameleon  functionality  which  supports  order  production  and 
transmission  depended  to  a  large  extent  on  the  specific  mission  activity  and  associated  time  pressure. 
In  general,  the  capability  was  seen  to  be  more  suited  to  slower  paced  phases  of  missions  than  to 
operations  such  as  a  warning  order  for  a  hasty  attack  and  counter  attack.  The  general  ability  to 
receive,  edit  and  transmit  a  warning  order  using  TBCS/Chameleon  was  seen  as  having  high  utility  and 
having  the  strong  potential  for  reducing  workload  and  eliminating  error.  Major  improvements  to  the 
actual  implementation  of  this  feature  were  seen  to  be  necessary  as  users  reported  that  editing  functions 
were  too  slow  and  cumbersome. 

Considerable  potential  was  seen  for  having  a  standard  overlay  accompany  the  order,  in  that  all 
recipients  would  have  identical  information,  thereby  reducing  transcription  errors  and  workload. 
Again,  users  found  the  actual  implementation  of  the  TBCS/Chameleon  functions  which  supports 
overlay  production  to  be  cumbersome  and  complicated,  a  view  which  we  would  support  based  on  our 
observations  of  individual  trial  participants  as  they  struggled  to  use  this  aspect  of  the  software.  While 
some  of  this  difficulty  could  be  attributed  to  insufficient  training,  this  would  not  account  for  much  of 
the  difficulties  encountered. 

Moreover,  and  perhaps  more  intractable,  users  reported  that  under  time  pressure  a  number  of 
difficulties  associated  with  order  production  and  transmission  arise.  Users  believed  that  while  the 
existing  HCI  and  functionality  would  support  order  production  for  a  planned  attack,  it  would  be  too 
slow  to  implement  in  a  hasty  attack  environment.  We  believe  that  these  comments  derive  more  from 
the  time  it  would  take  to  create  the  necessary  overlay  rather  than  the  belief  that  the  system  features 
have  potentially  low  utility  in  this  context. 

One  further  missing  element  from  the  order  production  and  distribution  process  that  should  probably 
be  addressed  as  TBCS/Chameleon  evolves  is  the  need  to  incoiporate  some  form  of  acknowledgement 
that  orders  have  been  received,  reviewed  and  comprehend. 
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4.4.6  Recommendations 

26)  Reorganise  system  features  that  support  creating  an  order  into  a  simpler  integrated  function  (i.e. 
the  plan  creation  process  should  not  be  executed  through  “overlays”).  This  will  help  to  decrease 
in  the  large  number  of  steps  now  required. 

27)  Simplify  the  “overlay”  feature.  Combine  control  measures,  co-ordinate  plans  and  orders.  Allow 
the  user  to  select  from  one  list  of  available  layers. 

28)  Provide  the  capability  to  show  the  “flanking  unit  picture”  in  the  overlay  menu.  This  provides  the 
user  (OC)  with  the  option  to  the  co-ordinate  plans  with  other  members  of  the  battle  group  (e.g. 
turn  on/off  other  combat  team  commanders’  battlefield  picture). 

29)  Allow  users  to  send  overlays  within  messages  more  quickly  and  easily  by  decreasing  the  task 
steps.  Integrate  this  with  recommendation  13. 

4.5  Access  to  resource  information 

A  number  of  system  features  can  be  grouped  into  this  category  including  friendly  units  Orbat,  unit 
information  TO&E,  operation  CEOI  and  Vetronics. 

The  consensus  of  the  group  was  that  the  friendly  units  Orbat  feature  had  good  utility  but  would  only 
be  used  by  limited  number  of  members  of  the  combat  team,  most  notably  squadron  and  company  level 
commanders.  For  these  individuals  the  tool  was  seen  to  be  most  useful  for  planning  activities.  A 
number  of  suggestions  were  received  concerning  improvements  that  could  be  made  both  to  the  utility 
of  the  feature  and  its  ease-of-use.  There  was  some  confusion  between  resource  information  to  be 
found  under  this  feature  and  that  found  under  friendly  unit  symbols.  It  appears  that  one  access  point  to 
draw  or  place  friendly  units  on  a  plan  and  access  information  about  their  status  and  capabilities  would 
be  more  useful. 

The  unit  information  feature  was  seen  to  be  desirable  only  by  the  21/C  to  assist  in  the  daily  collection 
of  logistics  information.  The  balance  of  the  combat  team  believed  the  feature  to  be  more  useful  to 
higher  levels  of  command.  Again,  limited  use  was  seen  for  the  TO&E  features  across  the  combat 
team,  and  users  believed  that  it  would  be  a  suitable  and  needed  tool  for  only  combat  team  commanders 
and  higher.  The  ability  to  build  queries  was  thought  to  be  useful  for  the  OC.  The  most  desirable 
improvement  to  the  current  functionality  was  seen  to  be  in  enhancing  the  database  to  reflect  actual 
resources  and  to  have  this  information  updated  at  regular  intervals. 

The  operation  CEOI  feature  received  consistently  positive  ratings  from  all  team  members  who  thought 
that  this  was  not  only  very  useful,  but  usable  right  now  in  its  existing  implementation  assuming  that 
additional  information  (see  user  comments  in  results  section  3.5.9)  and  security  features  were  added. 

4.5.1  Vetronics 

There  was  little  agreement  on  the  usefulness  of  the  Vetronics  capability  across  the  combat  team 
members.  In  spite  of  some  general  scepticism,  some  users  could  see  the  future  use  for  such  a 
capability  in  its  ability  to  provide  quick  situation  awareness  to  higher  units  of  capabilities  or  problems 
across  groups  of  vehicles  of  interest.  Users  believe  that  this  capability  would  be  greatly  enhanced  if 
Vetronics  were  integrated  with  all  vehicle  systems.  At  the  level  of  the  individual  vehicle,  Vetronics 
was  seen  to  be  more  useful  for  providing  an  early  warning  of  an  impending  problem,  rather  than  for 
providing  the  current  ongoing  status  of  vehicle  systems. 
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4.5.2  Recommendations 


30)  Orbat  database  should  reflect  current  unit/personnel/weapon  status. 

31)  Combine  Orbat  drawing  symbols  (currently  under  symbols,  friendly  units)  and  resource 
information  (currently  in  TO&E  in  VIPER).  Maximum  utility  will  be  achieved  through  providing 
one  place  for  friendly  and  enemy  Orbat  selection  for  status  information  or  for  selecting  the  symbol 
to  be  draw  on  the  map. 

32)  Provide  rapid  situation  awareness  of  aggregated  resource  status  through  graphic  means.  Provide 
the  option  to  expand  this  information  down  to  progressive  levels  of  detail,,  e.g.  provide  a  graphic 
which  shows  the  operational  effectiveness  of  a  Company  level  unit  -  the  user  could  expand  one 
level  down  to  rapidly  determine  the  status  of  the  Platoons,  which  again  should  be  represented  in 
graphical  format.  The  user  could  then  expand  one  level  further  to  see  text  based  detailed  such  as 
AFV  state  or  other  operational  status  information  (weapon  status,  weapon  symbol,  weapon  range, 
daily  battle  loss  etc.). 

33)  Provide  online  capability  to  reference  the  “Junior  General  Kit”  with  appropriate  hypertext  links. 

4.6  System  Options 

The  number  of  user  options  is  significant  and  provides  the  user  with  flexibility  in  setting  up  their 
preferred  configuration.  However,  it  is  not  clear  how  the  options  map  onto  the  specific  needs  for 
different  tasks  and  different  combat  team  personnel.  Because  this  feature  is  used  less  often  than  many 
others  (and  therefore  the  functions  are  more  quickly  forgotten)  emphasis  should  be  placed  in  making 
the  option  selections  self-explanatory  to  all  users. 

Users  should  be  provided  with  the  ability  to  rapidly  select  among  a  standard  set  of  pre-configured 
modes  of  operation  based  upon  an  analysis  of  user  requirements  and  context.  For  example,  an 
interface  and  tool  set  which  supports  combat  team  commander  level  planning  with  no  time  pressure  in 
a  hide,  would  not  be  suitable  in  the  context  of  a  section/crew  commander  in  a  hasty  attack.  Within  the 
pre-configured  modes,  some  degree  of  user  customisation  may  be  provided. 
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5 


Conclusions 


Overall,  the  current  functionality  of  TBCS/Chameleon  and  the  feature  set  employed  is  seen  to  serve 
the  needs  at  a  very  general  level  of  the  combat  team  across  the  range  of  combat  team  activities 
required  for  the  trial  scenario.  However,  for  individual  members  of  the  combat  team,  the  match 
between  perceived  utility  and  individual  need  varies  greatly. 

The  utility  trial  has  shown  that  there  are  a  number  of  major  problems  in  the  current  functionality  and 
these  are  addressed  in  point  form  below.  The  points  listed  below  concentrate  mostly  on  utility  issues. 
There  is  some  crossover  with  ease-of-use  issues  as  they  impact  upon  the  potential  utility  of  some 
system  features. 

5.1  Meeting  the  requirements  of  different  combat  team  positions 

•  At  present,  the  system  makes  no  attempt  to  provide  the  appropriate  functionality 
according  to  the  frequently  performed  tasks  associated  with  the  unique  role  of  different 
combat  team  members. 

•  The  risk  in  providing  a  comprehensive  feature  set  is  that  the  implementation  of  the  feature 
does  not  necessarily  address  the  specific  way  tasks  may  be  performed  differently  by 
different  combat  team  members. 

5.2  Meeting  the  requirements  of  different  operational  and 
mission  contexts 

•  Similar  tasks  performed  in  different  mission  contexts  may  require  different  system 
features  to  support  effective  task  execution.  The  specific  features  and  tools  provided  and 
means  of  user  interaction  need  to  be  tuned  to  the  specific  context  in  which  they  are 
performed. 

•  A  simplified  tool  set  needs  to  be  provided  for  tasks  conducted  under  time  pressure. 

•  Consideration  needs  to  be  given  for  the  specific  information  requirements  associated  with 
dismounted  activities. 

•  Map  features  need  to  be  implemented  selectively  to  meet  the  local  task  requirements. 

5.3  Speed  of  use 

•  The  system  features  need  to  be  organised  and  integrated  in  a  manner  which  minimises  the 
number  of  steps  required  to  be  taken  to  perform  a  task. 

•  The  actions  required  by  the  software  should  match  the  normal  sequence  and  logical  order 
in  which  tasks  are  done. 

•  The  interface  should  be  enhanced  to  allow  time  sensitive  tasks  (e.g.  sending  a  contact 
report)  to  be  performed  rapidly  and  simply. 

•  The  implementation  of  drawing  tools  needs  to  be  improved  to  allow  them  to  be  more 
effectively  used  to  support  rapid  planning  and  order  production. 
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5.4  Supporting  situation  awareness 

•  The  functionality  of  the  general  map  capability,  which  is  the  critical  core  component  of 
combat  team  tasks,  is  consistently  judged  to  be  worse  than  the  existing  capability.  Hence, 
improvements  in  the  design  of  this  feature  will  go  a  long  way  to  improving  system 
effectiveness. 

•  The  system  provides  little  support  for  rapid  awareness  for  new  information. 

•  The  system  impairs  the  integration  of  local  and  global  perspectives  and  spatial  integration 
of  information  (compared  with  what  can  be  achieved  now). 

•  Users  find  it  difficult  to  maintain  appropriate  map  situation  awareness  because  of  a  variety 
of  difficulties  associated  with  map  navigation. 

•  The  system  needs  to  provide  greater  support  for  visualising  future  event  status. 

•  Selective  and  additional  map  elements  will  need  to  be  added  to  support  situation 
awareness  of  terrain. 

•  The  system  has  the  potential  for  detracting  from  the  user’s  ability  to  operate  in  a  head-up 
mode. 

•  The  system  fails  to  provide  the  specific  support  (in  terms  of  the  appropriate  feature  and 
tool-set)  for  the  various  forms  of  situation  awareness  which  arise  in  different  mission 
contexts 

5.5  Supporting  communication 

•  The  system  fails  to  convey  the  urgency  currently  associated  with  certain  voice  messages. 

•  There  is  a  lack  of  feedback  on  whether  messages  have  been  received  and  understood. 

•  The  system  provides  insufficient  tools  to  support  effective  message  management  and  has 
the  potential  for  increasing  the  workload  associated  with  this. 

5.6  Beneficial  features  of  the  system 

•  The  provision  to  plot  unit  locations,  contacts  and  other  similar  information  (including  GPS 
data)  directly  on  the  map  provides  a  major  enhancement  to  situation  awareness  and  has  the 
potential  for  a  decrease  in  workload,  transcription  errors  and  communication  traffic. 

•  At  the  command  level,  the  system  enhances  the  effectiveness  and  accuracy  with  which 
orders  can  be  received,  prepared  and  transmitted. 

•  The  system  has  the  potential  for  streamlining  tasks  associated  with  the  collation  and 
integration  of  information  concerning  resources. 

•  The  system  enhances  the  maintenance  of  situation  awareness  and  communication 
effectiveness  when  the  user  is  required  to  change  command  and  control  systems,  for 
example  switching  vehicles. 

•  The  system  provides  good  support  for  enhanced  situation  awareness  in  planning 
operations. 

•  The  system  has  demonstrated  potential  for  reducing  voice  communication  and  allowing 
audio  channels  to  be  reserved  for  the  most  critical  information. 
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•  The  system  has  the  potential  for  reducing  a  number  of  error  sources  in  the  communication 
process. 

•  The  system  has  the  potential  to  enhance  situation  awareness  by  providing  access  to  unit 
level  information  concerning  capability/status. 

•  Users  can  see  considerable  potential  for  a  variety  of  TBC  S/Chameleon  based  decision  aids 
to  support  tactical  planning  tasks. 

5.7  Which  combat  team  positions  does  TBCS/Chameleon  serve. 

•  The  current  feature  set  appears  to  support  most  directly  the  requirements  of  the  OC  and 
possibly  2I/C.  This  conclusion  is  tempered  by  the  fact  that  the  trial  participant  who  played 
the  role  of  the  OC  represented  an  armour  perspective  only.  Hence  the  utility  of 
TBCS/Chameleon  to  support  senior  command  level  needs  from  an  infantry  perspective 
remains  unanswered  by  the  present  trial. 

•  All  members  of  the  combat  team  believed  that  some  of  the  TBC  S/Chameleon  features 
would  enhance  the  performance  of  some  of  their  tasks. 

•  As  currently  implemented,  the  feature  set  within  TBCS/Chameleon  has  progressively 
lower  utility  as  one  moves  down  the  chain  of  command  from  the  combat  team  commander 
to  crew/section  commander. 

5.8  Summary 

The  process  of  performing  a  concept  stage,  field  context,  design  review  of  TBCS  with  end  users  has 
proven  successful.  Recommendations  resulting  from  this  trial  can  be  implemented  in  to  the  next 
version  of  TBCS/Chameleon  software.  These  recommendations  concentrate  on  utility  issues  with  a 
secondary  focus  on  increasing  the  ease  of  use  of  some  features. 

The  following  factors  limit  the  predictive  value  of  the  results. 

•  the  low  level  of  experience  in  the  personnel  who  played  combat  team  roles  in  the  trial 

•  a  single  participant  at  each  combat  team  position  was  used  to  represent  the  entire  user 
population 

•  the  scope  of  the  scenario  does  not  capture  all  aspects  of  combat  team  operations 

The  user  review  process  should  continue  at  each  major  build  of  the  TBCS/Chameleon.  As  the 
development  moves  from  a  concept  based  development  to  a  fieldable  system  the  user  reviews  should 
move  from  utility  based  to  usability  based.  Tabletop  user  reviews  of  concepts  will  also  assist  with 
design  decisions  between  major  builds. 
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Annex  A  -  Scenario  Description 


This  table  describes  the  details  of  the  Advance  to  Contact  scenario  used  as  a  framework  for  conducting 
the  Chameleon/TBCS  Utility  Trial.  Explicit  details  for  every  action  of  each  position  cannot  be 
provided  because  participants  are  expected  to  role-play  during  each  stage  using  the  features  they  feel 
are  most  helpful.  Consequently  every  action  cannot  be  predicted.  Message  modes  used  by 
participants  depend  on  the  features  in  messaging  and  the  order/plan/overlay  sending  capability 
available  in  the  version  of  the  software  at  the  time  of  the  trial.  The  contents  of  the  four  columns 
contain  the  following  information: 

1.  Stage  of  scenario  and  electronic  trace  loaded  by  participants. 

2.  Grid  position  for  each  of  the  combat  team  participants  (T29, 139,  T29A,  13 9A,  T21, 13 1, 
Gl,  7  IF,  62  A,  62D,  Ell)  and  notional  combat  team  members  for  7  different  traces  using 
the  Cobden  map  (1:50  000,  Sheet  31  F/10,  Edition  4). 

3.  General  instructions,  prompts,  and  cues  for  participants. 

4.  Expected  Chameleon/TBCS  actions  of  participants. 


Scenario 

Segment 

Unit  Grid  Positions 

General  Instruction  to  Participants 

Expected  Participant  Actions 

1 .  Background 
(trace  #1) 

39  687432 

PARA  company 
combat  team  at 

651407.  O  Company 
combat  team  at 

675482  using  text 
“O”  9  at  702408. 

Participants  are  told  they  are  “dropped”  into  area  in  their 
position,  know  nothing  about  their  unit  or  combat  team. 

It  is  suggest  they  use  the  logistic  function,  map,  friendly 
Orbat,  message  in  their  tray  or  query  builder,  CEOI 
under  Operation,  Vetronics. .  .to  find  out  as  much 
information  about  their  unit  (subordinates  and  higher)  as 
they  can.  To  begin  the  are  told  to  load  trace  1. 

Participants  load  trace  1  containing  combat 
team  positions  and  “0”  position.  They  read 
a  paper  copy  of  the  overall  intelligence 
picture  (text  and  sketch).  They  read  the 
information  and  use  features  such  as 

ORBAT  to  uncover  information  about 
units  under  command.  Participant  will  also 
use  the  map  to  try  to  understand  the  ground 
and  general  tactical  situation. 

2.  Wng  0 
(trace  #1) 

As  above. 

Participants  are  sent  an  electronic  message  with  a  Wng 

O.  They  are  told  to  expect  a  Wng  O  and  are  expected  to 
respond  as  they  would  normally  by  going  through  battle 
procedures  (using  TBCS)  to  help  e.g.  detailed  map 
study  and  time  appreciation  etc.  Prompts  may  be 
required  to  have  participants  who  wouldn’t  normally 
send  a  Wng  O  do  so  to  exercise  the  system  feature. 

Participants  receive  a  message  containing  a 
Wng  O.  They  read  the  Wng  0,  conduct 
battle  procedures  and  then  create  and  send 
own  Wng  0. 

3.  OpO 
(trace  #2) 

62 A  at  670510 

PARA  662475  (same 
as  above  for  naming 
this  symbol) 

39  at  680475 

Participants  will  load  the  trace  and  will  receive  an 
electronic  Op  O.  They  are  told  they  will  receive  the 

CO’s  Op  O  and  are  expected  to  create  their  own  as  they 
might  in  the  field  and  then  get  to  the  position  where  they 
would  send  this  out  using  a  text  message  and  trace. 

Participant  opens  trace  with  CO’s 
intentions  and  then  creates  own  Op  0. 
Participant  creates  own  Op  0  using  trace 
and  messaging.  May  create  new  order  or 
may  change  current  map. 

4.  Update 

Wng  0 
(trace  #2) 

As  above. 

Participants  are  told  they  are  getting  close  to  H-hr  and 
may  want  to  check  last  minute  details  of  logistics  or 
plans  and  then  make  an  update  to  their  Wng  O  or  Op  O. 
They  should  take  a  couple  of  minutes  and  use  TBCS  to 
do  last  minute  checks. 

Participant  use  TBCS  messaging  to  update 
Wng  0  or  Op  0  if  required  and  may  need 
to  search  for  any  last  minute  update  of 
logistics  or  unit  locations. 
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Scenario 

Segment 

Unit  Grid  Positions 

General  Instruction  to  Participants 

Expected  Participant  Actions 

5.  Move  to 

21  at  691448 

Participants  are  told  that  H-Hr  is  really  close  and  they 

Participants  will  move  their  units  along  the 

Assembly  Area 

22  at  689447 

are  now  going  to  move  to  the  assembly  area/attack 

road.  They  may  perform  map  navigation 

(trace  #3) 

23  at  688445 

position.  To  do  this  they  should  take  about  10-15 

and  will  certainly  study  the  map  in  detail  to 

T29 At  687444 

minutes  to  move  their  own  unit(s)  along  the  road 

discriminate  units. 

T29A 686443 

(slightly  north  of  their  position  bearing  east  northeast 

24  at  685442 

then  north  northwest)  in  small  increments  -  they  may 

31  at  682440 

get  ahead  of  other  units  or  get  mixed  up  but  try  to 

32  at  680438 

maintain  a  realistic  pace  -  keep  an  eye  out  for 

33  at  678434 

messaging.  They  are  told  to  take  about  15  min.  to  get  to 

39  at  679433 

G1  at  678436 

the  attack  position. 

39A  at  680432 

A  cue  is  given  to  T21  -  T21B  damaged  by  mine  while 

Ell  at  658516 

moving  to  LD  -  at  Gr  695456  (stream)  T21B  loses  track 

T21  will  receive  cue  card  and  will  initiate 

71F  at  695455 

and  blocking  road.  Note  this  cue  should  be  give  before 

messaging  indicating  the  current  situation. 

62A  670510 

the  advancing  column  crosses  the  stream  at  the  above 
grid  reference  point. 

6.  At  LD 

As  above. 

The  OC  is  instructed  to  place  units  i.e.  indicate  to  the 

The  OC  should  work  out  positions  for 

(trace  #3) 

combat  team  where  they  should  locate  at  the  LD,  by 

attack  using  paper  or  the  current  map  and 

way  of  message.  Other  participants  are  told  they’re 

send  out  messages  to  the  combat  team 

close  to  the  LD  and  wait  for  final  message  instructions. 

indicating  the  desired  formation  for  the 

When  they  receive  the  OC’s  message,  move  to  the 
required  position  and  wait  for  a  verbal  indication  of  h- 
hr. 

advance. 

While  the  OCs  are  planning  the  attack  formation, 
participant  are  told  to  try  the  options  (if  they  haven’t 
already)  at  the  bottom  of  their  screen  for  message 
waiting,  current  cursor  position,  current  date,  active  unit 
position,  current  cursor  state  and  current  map  recenter 
state. 

Participants  will  exercise  status  display 
options  at  the  bottom  of  the  screen. 

Participants  then  move  themselves  into 
position. 

7.  Move  to  first 

As  above. 

H-Hr  is  indicated  and  participants  are  instructed  to 

Participants  move  symbols  and  wait  for 

objective, 

advance  their  unit/vehicles  at  a  realistic  pace  up  to  the 

contacts. 

location 

first  report  line  and  be  prepared  for  contacts. 

awareness. 

(trace  #3) 

Participants  will  be  queried  about  their  own  position  and 

Messaging  will  take  place  to  indicate  to  the 

of  everyone  else. 

combat  team  the  locations. 

A  cue  is  given  to  a  participant  to  indicate  that  they  have 

Messaging  will  take  place  to  indicate  to  the 

hit  a  minefield.  1 

details  of  a  minefield. 

1  Details  of  the  cue  i.e.  time,  grid  reference  and  individual  participant  depend  on  the  circumstances  of  the  free 
play  scenario. 
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Scenario 

Segment 

Unit  Grid  Positions 

General  Instruction  to  Participants 

Expected  Participant  Actions 

8a.  First  Set  of 

21  at  656520 

Participants  are  told  to  load  up  trace  4  and  are  asked  to 

Participants  load  new  trace  and  move 

Contacts 

22  at  661523 

continue  their  advance.  They  should  expect  an 

symbols  and  wait  for  contacts. 

(trace  #4) 

23  at  667528 

encounter  with  the  enemy.  They  are  asked  to  respond  to 

When  they  receive  a  cue  they  will  make 

T29  at  666524 

contacts  as  they  normally  would  using  TBCS.  They  are 

contact  reports  similar  to  below: 

T29A 668523 

asked  to  try  to  move  their  units  in  a  tactically  realistic 

24  at  670524 

fashion.  They  should  take  about  20  minutes  to  go  to  the 

1)  62 A  -  Gr  624540  2x  brdm’s-2 

31  at  661519 

next  report  line. 

moving  southeast  along  spade  route  at 

32  at  674522 

high  speed  -  observing. 

33  at  676520 

Provide  five  contact  cues  to  various  participants 

2)  62D  -  Gr  65 1 550,  section  minus  with 

39  at  660518 

examples  are: 

BMP-2  dug  in  at  defile.  Moving 

G1  at  669521 

northeast. 

39A  at  658516 

1)  . .  .you  can  see  2x  BRDM’s-2  moving  southeast 

3)  7 IF  -  Gr  624541  2x  brdm’s-2  moving 

Ell  at  661541 

along  spade  route  at  high  speed  at  and  are 

southeast  along  spade  route  - 

71F  at  653518 

observing. 

observing 

62A  656545 

4)  T29A  -  Gr  625541  3x  brdm’s-2 

62D 660545 

2)  . .  .you  see  a  section  minus  with  BMP  2  dug  in  at 

moving  south  along  spade  route  at 

PARA  coy  at  635497 

defile  at  Gr  656545  and  you  are  moving  northeast 

high  speed  -  observing 

etc. 

5)  G1  -  Gr  626541  lx  BMP  moving 

south  along  spade  route  -  observing 

The  contacts  above  should  appear  to  be 
multiple  reports  of  the  same  enemy  causing 
the  OC  to  make  some  contact  report 
consolidation  decisions. 

9.  Show 

21  at  631540 

Participants  are  told  to  load  up  trace  5.  They  are  told 

Participant  will  load  the  new  trace  that 

contacts 

22  at  636542 

they  will  be  experiencing  typical  battlefield  situations 

shows  new  positions  of  the  combat  team. 

destroyed,  Show 

23  at  643554 

and  report  traffic.  They  should  try  to  respond  to  the 

PARA  is  falling 

T29  at  632542 

situations  in  a  typical  fashion  but  using  TBCS.  If  a 

behind. 

T29A  at  650560 

particular  type  of  message  is  not  available,  use  text 

(trace  #5) 

24  at  648558 

message.  Keep  moving  units  at  a  real  battlefield  pace 

31  at  635540 

32  at  647555 

and  take  1 5  min.  to  get  to  the  next  report. 

33  at  650553 

Prompt  a  team  member  to  report  destroyed  enemy  in 

39  at  637541 

previous  contacts. 1 

A  participant  will  send  a  message  to 

G1  at  650551 

indicated  the  enemy  destroyed  (reece  and 

39A  at  644553 

Ell  at  647576 

Prompt  to  T21  to  report  T21C  destroyed. 

destroyed  section). 

71F  at  630538 

62A  646577 

62D 646578 

Send  a  message  from  9  requesting  a  sit  rep. 

A  participant  will  report  the  loss  of  2 1C. 

PARA  coy  at  635497 

Prompt  7 IF  to  give  contact  rep.  (BMP-2  moving  NW 

Gr  622522  at  high  speed  along  diamond  route). 

29  this  7 IF  contact... 

A  cue  will  be  given  to  show  PARA  Coy,  a  flanking  unit, 
is  falling  behind  e.g.  29,  this  is  9,  call  sign  4  (para  coy) 
is  delayed  dealing  with  difficult  with  enemy  at  position 

The  OC  may  send  messaging  after  he  or 

Gr  630504 

she  sees  PARA  Coy  is  falling  behind. 

1  Details  of  the  cue  i.e.  time,  grid  reference  and  individual  participant  depend  on  the  circumstances  of  the  free 
play  scenario. 
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Scenario 

Segment 

Unit  Grid  Positions 

General  Instruction  to  Participants 

Expected  Participant  Actions 

10.  More 

21  at  613569 

Participants  are  told  to  load  trace  6.  Participants  are 

Participant  will  load  new  trace. 

contacts  leading 

22  at  615572 

asked  to  keep  moving  their  units  as  they  think  they 

to  the  necessity 

23  at  622578 

would  on  the  battlefield  and  go  to  the  next  report  line  in 

of  a  Hasty 

T29  at  624580 

the  next  15  minutes.  They  can  expect  some  more 

Attack  plan. 

T29A  at  612567 

serious  problems  regarding  enemy  during  this  and  the 

(trace  #6) 

24  at  630573 

next  couple  of  segments. 

31  at  615567 

32  at  632578 

Prompt  62A  (recce)  to  make  a  contact  report  of  dug  in 

62A  will  make  a  contact  report. 

33  at  633574 

enemy  PI  along  north  bank  of  stream  Gr  613593, 

39  at  627581 

G1  at  605566 

(quail)  trying  to  work  to  n  along  railroad  track. 

39A  at  613568 

Prompt  62A  (recce)  to  reference  last,  can  observe 

Ell  at  632588 

2xBMP-2s  and  lxT72s  and  a  BRDM-2. 

62A  will  follow  up  the  previous  contact 

71F  at  610567 

report.  This  should  indicate  to  the  OC  the 

62A  at  632588 

Prompt  OCs  to  provide  sit  rep.  The  OC  may  need  a 

need  for  a  hasty  attack. 

62D  at  633589 

prompt  to  create  a  hasty  attack  plan.  All  participants  are 
asked  to  create  their  own  hasty  attack  plan  using  send 

OC  should  provide  a  sit  rep.  All  should 

overlay  feature  to  exercise  the  feature. 

realize  the  need  for  a  hasty  attack  and  will 
create  a  plan  for  this. 

1 1 .  Distribute 
the  plan  for  the 
Hasty  Attack, 

21  at  595589 

22  at  600583 

23  at  592585 

Participants  are  told  to  load  trace  7  that  shows  everyone 
in  position  for  the  hasty  attack.  They  will  be  asked  to 

Participant  will  load  the  plan  for  the  hasty 
attack  and  use  TBCS  to  perform  any 
preparation  they  have. 

preparation  and 

T29  at  595586 

prepare  to  conduct  this  attack. 

counter  attack. 

T29A  at  622588 

All  receive  the  contact/sit  rep  indicating 

Plan  and  move 

24  at  619588 

All  participants  receive  a  message  from  9  e.g.  2  this  is 

the  counter  attack. 

to  defensive 

31  at  597585 

9er,  from  H6,  enemy  column  detected  moving  south 

(trace  #7) 

32  at  595583 

long  diamond  route.  Lead  element  just  passing  Gr 

33  at  594582 

617646.  Column  contains  6xT72.  Note  this  cue  should 

39  at  597583 

be  given  before  participants  begin  the  hasty  attack  i.e. 

G1  at  622593 

39A  at  593583 

within  the  first  few  minutes  of  loading  the  trace. 

Ell  at  636567 

This  should  cause  OC  to  want  to  reorient  units  to  face 

All  participants  will  create  and  send  plans 

71F  at  589582 

62A  630590 

present  threat  prior  to  conducting  the  hasty  attack. 

for  a  hasty  defense. 

62D  631592 

All  participants  are  cued  into  the  situation  and  everyone 

PARA  coy  at  585542 

will  be  asked  to  create  a  hasty  defense  plan  and  send  to 
the  person  opposite  to  them  in  the  room  to  exercise  the 
feature. 

12.  Executed 

21  at  625581 

Participants  are  told  to  load  trace  8.  Participants  will  be 

Participant  loads  new  trace  to  see  new 

defensive  plan 

22  at  626580 

told  the  defensive  was  successful  and  that  they  will  have 

positions  of  all. 

and  conduct  of 

23  at  635579 

to  conduct  the  hasty  attack.  They  should  move  their 

hasty  attkack. 

T29  at  639579 

units  and  expect  more  messaging.  As  they  move  into 

Employment  of 

T29A  at  627578 

position  and  conduct  the  attack  they  should  think  about 

Participants  re-conduct  the  attack  on 

engineering 

24  at  640583 

how  they  would  use  TBCS  in  a  real  situation.  For 

Foresters  Falls. 

resources. 

31  at  637580 

example,  when  the  infantry  dismounts,  should  every 

(trace  #8) 

32  at  630576 

section  commander  have  a  portable  TBCS,  every  PI 

33  at  631573 

39  at  641580 

commander,  every  coy  commander  etc. . . 

G1  at  633572 

A  participant  is  prompted  to  give  a  sit  rep  for  destroyed 

39A  at  625576 

enemy  e.g.  you  can  see  5xt72  destroyed  by  fire  on  road 

Participants  send  update  message  of  the 

Ell  at  636576 

to  the  north. 1 

enemy  situation  to  all. 

71F  at  624581 

62A 623584 

A  participant  is  prompted  to  give  indication  of  railway 

62D  623585 

bridge  at  622595  destroyed. 

The  message  sent  with  the  bridge 
destroyed  information  should  cause  some 
assessment  of  engineering  resources  by  the 
OC.  e.g.  29  this  is  El  1 A  (eng  Recce) 
Brooms  Creek  is  unfordable,  good  bridge 
site  at  Gr  596582  in  the  low  ground. 

1  Details  of  the  cue  i.e.  time,  grid  reference  and  individual  participant  depend  on  the  circumstances  of  the  free 
play  scenario. 
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Scenario 

Segment 

Unit  Grid  Positions 

General  Instruction  to  Participants 

Expected  Participant  Actions 

13. 

Consolidation 
(trace  #8) 

As  above. 

Participants  are  told  they  are  going  to  a  consolidation 
phase.  They  should  follow  SOP  and  move  into  the 
positions  they  think  they  would  adopt  and  conduct  the 
normal  routines.  They  should  pay  special  attention  to 
logistics  features  and  where  they  think  the  potential 
benefits  of  TBCS  are  for  this  activity. 

Participants  use  messaging  and  resource 
features  to  create  post  battle  and  daily 
logistic  reports. 
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Annex  B  -  Operation  order 


OP  Order  for  Chameleon  TBCS  Utility  Trial 

24  Nov  97 

CFM  NOTES  TO  00S  GIVEN  BY  CO  RCR  AT  24  0900R  NOV  97 
Refs:  A.  CANADA  Sheet  31  F/10  (COBDEN)  Edn  4,  1:50,000 

Time  Zone:  Romeo  (local) 

SITUATION 


En  Forces.  As  per  Int  Brief. 

Friendly  Forces. 

Div  Comd’s  Intent. 

1)  Purpose.  To  support  the  Coips  Comd’s  plan  of  destroying  the  cbt  capability  of  the 
KRASNOVIAN  forces  in  BH;  thereby  removing  the  KRASNOVIAN  military  threat  and 
creating  conditions  for  a  lasting  peace. 

2)  Method.  1  Can  Div  is  currently  deployed  with  1  CMBG  fwd,  the  remainder  of  the  Div 
indepth  with  2  CMBG  RIGHT  and  5  CMBG  LEFT.  1(CA)  Div  will  adv  to  the 
KRASNOVIAN  Border  in  three  Ph: 

Ph  1 .  2CMBG  will  adv  to  the  Petawawa  River  while  5  GMBC  will  secure  the  Div 
LEFT  flank  by  occupying  the  CONSTANT-CLEAR-GOLDEN  and  DORE  LAKES 
region. 

Ph  2.  1CMBG  will  conduct  an  aslt  river  crossing  of  the  PEAWAWA  River  to  destroy 
the  remnants  of  3 1  and  37  MRR. 

Ph  3.  5  GBMC  will  pass  through  2  CMBG  and  1  CMBG  and  exploit  to  the  KRASNOVIAN 
border. 

3)  End  State.  To  have  destroyed  all  elements  of  the  KRASNOVIAN  military  in  1  (CA)  Div 
sector  up  to  the  KRANOVIAN  Border. 

Comd  2  CMBG’s  Intent. 

1)  Purpose.  To  secure  the  SOUTH  hank  (or  both  banks)  of  the  PETAWAWA  river  in 
preparation  to  sup  crossing  by  1  CMBG. 

2)  Method.  Through  rapid,  aggressive  action  of  fr  forces  190  MRR  and  491  lndep  TB  will  be 
destr  and  the  SOUTH  bank  of  the  PETAWAWA  River  will  be  secured.  Deep  and  close  battles  will  be 
fought  concurrently  through  the  aggressive  use  of  ARTY,  CAS  and  AH  fires;  thus  keeping  the  en  off- 
balance  throughout.  2  CMBG  will  advance  to  the  PETAWAWA  River  in  three  phs  with  the  KRH  BG 
in  res  throughout: 

Ph  1.  2CMBG  will  adv  with  two  BG  up  with  RCD  LEFT  and  3  RCR  RIGHT.  RCD  will 
secure  Obj  SNAKE  and  3  RCR  will  secure  Obj  BIRD. 

Ph  2.  On  order  advance  and  secure  Obj  CAT  with  3  RCR  or  RCD. 
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Ph  3. 1  RCR  will  pass  through  the  RCD  and  3  RCR  and  secure  Obj  DOG. 

3)  End  State.  2  CMBG  End  State  will  see  1  RCR  secure  on  the  SOUTH  hank  of  the 
PETAWAWA  River  prepared  to  sp  1  CMBG  Assault  Water  Crossing.  The  three  remaining 
BGs  will  remain  in  depth  prepared  to  sp  the  fwd  passage  of  1  CMBG  and  5  GBMC.  All  en 
within  the  2  CMBG  sector  will  have  been  destroyed  or  captured. 

Six  CAS  sorties  have  been  alloc  to  2  CMBG  daily  from  0700  hrs  25. 

Atts  and  Dets.  See  Gp  and  Task  Matrix 

MISSION 

TO  SECURE  Obj  CAT  by  2000  hrs  25  Nov. 

EXECUTION 
CO’s  Intent. 

Purpose.  To  seize  Obj  BIRD  with  a  view  to  either  effecting  adv  to  secure  Obj  CAT,  or 

sp  adv  of  RCD  to  Obj  CAT;  then,  deploy  for  sp  of  subsequent  adv  of  1  RCR  to  Obj  DOG. 

Method.  3  RCR  will  sp  the  Comd’s  plan  as  follows: 

Ph  1.  We  will  adv  with  two  cbt  tms  up  PARA  Coy  Cbt  tm  LEFT  and  B  Sqn  cbt  tm 
RIGHT  as  the  main  effort  with  O  Coy  Cbt  Tm  in  Reserve  clearing  CLUB  Rte  and 
securing  the  LD.  Recce  PI  well  fwd  and  our  right  flank  covered  by  TUA.  We  will 
destroy  all  enemy  encountered.  We  will  cut  off  escape  rtes  so  that  we  don  not  face  the 
same  en  twice.  We  must  maintain  momentum  throughout  and,  therefore,  be  careful 
not  to  allow  the  enemy  to  force  us  to  deploy  our  forces  at  every  encounter  and  thus, 
delay  our  advance.  The  end  state  for  Ph  1  will  see  PARA  Coy  Cbt  Tm  secure  on  Obj 
ROBIN;  B  Sqn  Cbt  Tm  secure  on  Obj  QUAIL;  O  Coy  Cbt  Tm  will  secure  LD  and  be 
prepared  to  continue  the  adv  on  order;  Recce  conducting  area  recce  of  peninsula  to  N 
of  BIRD. 


Ph  2.  On  order,  O  Coy  Cbt  Tm  as  the  main  effort  will  continue  to  adv  to  Obj  CAT.  O 
Coy  Cbt  Tm  will  estb  The  Bridgehead  into  PEMBROKE  and  B  Sqn  Cbt  Tm  will  pass 
through  O  Coy  Cbt  Tm  and  continue  the  FIBUA  Battle  as  the  BG  Main  Effort.  PARA 
Coy  Cbt  Tm  will  move  to  the  Northwest  of  CAT  and  act  as  Cut  Off  and  Flank 
Security.  O  Coy  Cbt  Tm  will  then  remain  in  res. 

Ph  3.  All  elements  prepared  to  support  the  fwd  passage  of  1  RCR  onto  Obj  DOG.  O 
Coy  Cbt  Tm  will  be  the  Main  Effort  for  Reorg/Reconstitution  in  anticipation  of  being 
brought  fwd  to  sp  1  RCR. 

End  State.  B  Sqn  Cbt  Tm  secure  on  Obj  CAT,  PARA  Coy  Cbt  Tm  estb  to  the 
NORTH  WEST  of  CAT,  and  O  Coy  Cbt  Tm  in  res.  All  elements  prepared  to  assist  1 
RCR  adv  and  atk  onto  Obj  DOG. 

Gp  and  Tasks.  IAW  Gping  and  Task  Matrix. 
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Coord  Instr. 


Timings: 


Regrouping  complete 
Back  brief  to  CO 
H  Hr 

Ph  1  complete 
Ph  2  commences 
Ph  2  complete 
Ph  3  commences 
Ph  3  complete 


2100  hrs  Nov  24 
2300  hrs  Nov  24 
0630  hrs  Nov  25 
1300  hrs  Nov  25 
on  order 
1700  hrs  Nov  25 
on  order 
2000  hrs  Nov  25 


Report  Line,  Rtes,  Bdrys,  Contact  Pts.  As  per  trace. 


By-pass  Policy.  En  within  sector  will  surrender  or  be  destr.  No  by-pass  by  BG;  however,  CO 
3 


RCR  may  auth  lead  Cbt  Tms  to  bypass  sect  and  smaller  en. 

Fire  Plan.  As  per  initial  briefs,  CFSP  to  concentrate  on  FD  and  Obj  BIRD,  Cbt  Tm  CFSP  to 
BG  FSCC  NLT  2300  hrs  25  Nov.  BG  CFSP  TBI  2100  hrs  24  Nov. 


Recce.  No  recce  fwd  of  NEW  SHOE  before  0530  25  Nov. 

Engineers.  OC  24  Fd  Sqn  to  coord. 

NBC.  TOPPLOW. 

Pri  of  Tgts.  Comd  and  Con,  Tanks,  APC’s. 

Limit  of  Exploitation.  For  Ph  1  will  be  BASS  DRUM.  For  Ph  2  and  3  it  will  be  RED 
SANDAL. 

Air  and  Avn.  Comd  2  CMBG  will  be  using  all  air  and  avn  in  the  deep  battle.  Req  of  air  and 
avn  for  high  pri  tgts  are  to  be  submitted  through  BG  HQ. 

Open  Fire  Pol.  Prior  to  H-Hr  self-defence  only.  At  H  Hr  all  ident  en  not  in  the  process  of 
surrendering  are  legitimate  tgts.  Vehs  remain  legitimate  tgts  if  wdr.  Tps  remain  legitimate 
tgts  if  wdr  with  wpns.  En  which  surrenders  or  wdr  without  wpns  are  not  legitimate  tgts  and 
will  not  be  engaged. 

SERVICE  SUPPORT 

Resup.  No  add  resup  until  completion  of  Ph  3.  Ea  coy  to  maint  24  hrs  cbt  sup. 
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COMMAND  AND  SIGNALS 


Altn  Comd.  DCO  then  OC  O  Coy 
Locs: 

3  RCR  G  CP  Main  initially  at  GR  xxxxxx  UF 

2  CMBG  Main  at  ARNPRIOR  UR 
CO’s  TAC  with  main  effort. 

Radio  Silence. 

Radio  Silence  remains  in  effect. 

Broken  on  contact  by  cbt  tms. 

Lifted  by  3  RCR  BG  HQ  only 

3  RCR  G  CP  Main  initially  at  GR  UF 
2  CMBG  Main  at  ARNPRIOR  UR 


Frequencies.  ALL  TBC+ 

BG  Comd  Pri3010 

N  Coy  Cbt  Tm  Comd  Pri  3210 

PARA  Coy  Cbt  Tm  Comd  Pri  3360 

BG  Fire  Support  Net  Pri  4490 


Alt  3420 
Alt  3610 
Alt  3745 
Alt  3535 


Code  Words 


Ser 

Code  Word 

Meaning 

Issued  By 

a) 

b) 

c) 

d) 

1 

OAK 

LD  Crossed 

All 

2 

ELM 

Obj  SNAKE  Secure 

1  RCR 

3 

BIRCH 

Obj  BIRD  Secure 

3  RCR  HQ 

4 

BEECH 

Obj  QUAIL  Secure 

B  Sqn  Cbt  Tm 

5 

COCONUT 

Obj  ROBIN  Secure 

PARA  Coy  Cbt  Tm 

7 

HEMLOCK 

Commence  Ph  2 

3  RCR  HQ 

8 

MAPLE 

Obj  CAT  Secure 

3  RCR 

9 

PINE 

Obj  Commence  Ph  3 

3  RCR  HQ 

10 

ALDER 

Obj  DOG  Secure 

1  RCR 

ACK INSTRS:  Ack 
Author: 

LCol 

Commanding  Officer 

Authentication 

Capt 
Ops  O 
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Annex  A  -  Gping  and  Task  Matrix 
Annex  B  -  Trace 
Annex  C  -  Int  Brief 

DISTR 

PARA  Coy 
N  Coy 
OCoy 
Q  Coy 
R  Coy 

HQ  2  CMBG 
Recce  Sqn 
2  RCHA 
RCD 

1  RCR 

2  CER 
CO 
OA 
OB 
FSCC 
Spare 
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ANX  A 


GP  AND  TASK  MATRIX  -  CFM  NOTES  OPS  01 


PARA  Coy 

B  Sqn 

OCoy 

Recce 

Atts  and  Dets 

Under  Comd 

A  Sqn  RCD 

Under  Comd 

3  Dets  TUA 

Under  Comd 

Det  TUA 

Under  Comd 

A  Sqn  RCD 

B  Sqn  RCD 

C  Sqn  RCD 

N  Coy  3  RCR 

C  Sqn  RCD 

In  Sn 

Pnr  PI 

In  Sp 

Tp  armd  engr 

In  Sp 

To  enar 

In  Sp 

Engr  Recce  Dct 

In  Sp 

Armd  Engr  Tp 

Tp  fd  engr 

Alloc 

G2 

Alloc 

G1 

Alloc 

BC/FOOs  F  Bty 

2  RCHA 

Tasks 

Tasks 

Tasks 

Tasks 

Ph.  1 

LEFT  fwd  Cbt  Tm 
Destroy  en  between 
NEW  SHOE  and 

BASS  DRUM 

Be  prep  to  man 

Contact  Pt  CX 

Secure  Obj  ROBIN 

Ph.  1 

RIGHT  twd  Cbt  Tm 
Destroy  en  between 
NEW  SHOE  and 

BASS  DRUM 

Secure  Obj  QUAIL 

Ph.  1 

Secure  LD 

Clear  CLUB  rte 

BG  Reserve 

Ph.  1 

Provide  Recce  well 
fwd  on  both  axis 

Est  Obj  BIRD 

Arty  Tasks 

F  Bty, 

2  RCHA  DS 

3  RCR 

Tp  89  Bty 

DS  3  RCR 

Engr  Tasks 

Mobility  tasks  on  sp 
of  fwd  Cbt  Tms 

Rte  clearance  of 

CLUB  rte 

Clearance  of  booby 
traps  within  built  up 
areas 

Recce  Tasks 

Close  fwd  recce  in  sp 
of  cbt  tms 

TUA  Tasks 

Long  rge  dir  fire  sp 

Cbt 

Ph.  2 

Be  prepared  to  sp  C 
Sqn  Cbt  Tm  or  RCD 
adv  to  CAT 

Ph.  2 

Be  prepared  to  sp  C 
Sqn  Cbt  Tm  or  RCD 
adv  to  CAT 

Ph.  2 

Be  prepared  to  adv  to 
CAT  or  sp  RCD  adv 
to  CAT 

Ph.  2 

Be  prepared  to  sp  C 
Sqn  Cbt  Tm  or  RCD 
adv  to  CAT 

Ph.  3 

Be  prepared  to  sp 
fwd  passage  of  1 

RCR 

Ph.  3 

Be  prepared  to  sp 
fwd  passage  of  1 

RCR 

Ph.  3 

Be  prepared  to  sp 
fwd  passage  of  1 

RCR 

Ph.  3 

Be  prepared  to  sp 
fwd  passage  of  1 

RCR 
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ANX  C  -  Intelligence  Annex  to  3RCR  Ops  001 
Summary  of  En  Situation 

The  Krasnovian  1  CAA  advance  on  three  axis,  towards  parry  Sound,  Huntsville  and  Pembroke, 
forcing  the  Multi-National  Bde  to  Wdl  back  to  the  Carlatian  Border.  The  1  CAA’s  immediate  obj  was 
to  seize  the  Ottawa  valley  as  far  south  as  Amprior,  but  the  Zepher  dominated  Barrian  Federation  Army 
has  been  holding  the  Krasnovians  in  the  Muskrat  lake  area  since  2  Nov.  We  believe  that  their  final 
Obj  was  to  retake  all  of  the  BH,  incorporating  it  into  the  greater  Krasnovia. 

The  Krasnovians  launched  an  attack  on  BH  on  1 6  Oct.  1  CAA  of  the  KRA  advanced  on  three  axes.  In 
the  west,  79  MRD  advanced  south  along  Hwy  69  to  Parry  Sound.  94  MRD  moved  along  Hwy  1 1  to 
Huntsville.  These  two  Divs  are  holding  in  loc  despite  the  lack  of  BH  resistance. 

In  the  east  80  MRD  advanced  southeast  along  the  Ottawa  valley  using  Hwy  17  as  the  main  axis.  37 
Tk  Div  is  in  the  North  Bay  area  and  appears  to  be  oriented  t  advance  along  Hwy  17.  The  assessed 
immediate  obj  is  the  Town  of  Amprior. 

The  80  MRD  is  deployed  with  two  MRRs  up  (3 1st  and  27st  east)  with  the  remnants  of  the  190  MRR 
and  the  91  TR  on  a  fwd  screen  and  covering  force  posns  as  far  south  as  Mclarens  Settlement  Gr  UR 
678519.  3 1st  and  37th  MRRs  have  been  preparing  def  posns  for  the  past  36  hrs,  their  strength  is  est  to 
be  approx  50%. 


Air  Superiority:  The  80  MRD  is  capable  of  local  A/S  for  up  to  30  mins 
NBC:  The  En  has  NBC  capable  and  may  use  persistent  agents  to  deny  mobility. 
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TBCS  /  Chameleon 


User  Review 


Hummsystems'  TBSC/Chameleon  Utility  Trial  Report  Annex  C-  2 


TBCS  /  Chameleon 


User  Review 


1 .  Personal  Information 


All  the  data  you  provide  will  be  kept  in  confidence  but  we  need  background  information  to  determine  how  different  users  have 
different  requirements  for  system  features. 


1.1  Name  &  Rank: 
1.3  History: 


1.2  Years  in  Canadian  Forces 


History  in  Combat  Team  and  Other  Appointments 

Unit 

Position  in  Cbt  Tm 

Time  in  position  (yr,mo) 

e.g.  12  PI  D  Coy 

1RCR 

PI  Comd 

1  yr  9  mo 

1.4  Computer  Experience  (typical  usage): 


Desk  top  PC’s 

□Never 

□Occasional 

□Frequent 

Lap  top  PC’s 

□Never 

□Occasional 

□Frequent 

Windows  3.1 

□Never 

□Occasional 

□Frequent 

Windows  95 

□Never 

□Occasional 

□Frequent 

Mac/Apple  products 

□Never 

□Occasional 

□Frequent 

Keyboarding 

□Never 

□Occasional 

□Frequent 
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TBCS  /  Chameleon 


User  Review 


3.  Position  for  this 

T29  ^  139  ^  T29A  ^  I39A  ^ 

Gil  - 

Scenario 

T21* 

131  * 

62  A  * 

62  71F  * 

2.  Name  and  Rank 

Last  name 

Initials 

Rank 

IJtilitv 

Usability 

4.  TBCS  Features 

Allow  you  to 

Desirable  feature 

Improvement  over  the 

Ease  to  use 

Ease  to  use  in  the 

operate  effectively  ? 

Not  at  Helps  Major 

All  Somewhai  Benefit 

final  system  ? 

Not  at  Somewhat 
All  Desirable 

Highly 

Desirable 

current  capability  ? 

Much  Neither  Much 

worse  worse/better  Better 

Very 

Difficult 

Here? 

Acceptabl 

e 

Very 

Easy 

vehicle  /  field  ? 

Very  Acceptabl  Very 

Difficult  e  Easy 

1 .  Overlays  -  control  measure 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

2.  Overlays  -  coord  plans 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

3.  Overlays  -  orders 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

4.  Messaging  -  general 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

5.  Messaging  -  Contact  report 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

6.  Messaging  -  Wng  O 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

7.  Messaging  -  send  overlay 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

8.  Symbols  -  fr  units  (Orbat) 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

9.  Symbols  -  en  unit  editor 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

10.  Map-  fonnat  (scale,  detail,  etc.) 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

1 1 .  Map-  navigation  (pan,  zoom 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

12.  Map-  drawing  (plans,  etc.) 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

13.  Unit  Information  -  (i.e.  right 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

14.  TO&E  -  Orbat 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

15.  TO&E  -  resources 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

16.  TO&E  -  Info  query  builder 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

17.  Operation  -  CEOI 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

18.  Message  state  (red,  green. 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

19.  Cursor  posn  indication 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

20.  Date  time  indication  (options) 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

21 .  Active  unit  posn 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

22.  Cursor  state  (single,  multi,  etc.) 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

23.  Map  recentre  mode  (fixed  etc.) 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

24.  Vetronics 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

25.  System  -  options 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

26. 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

27. 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

28. 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

29. 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

30. 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 
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Annex  D  -  Discussion  of  validity  issues  concerning 
the  trial 


One  check  that  can  be  performed  to  assess  the  validity  of  the  questions  that  were  asked,  is  to  look  at 
how  they  are  inter-correlated.  For  example,  when  trial  participants  rate  the  utility  of  a  system  feature 
in  terms  of  its  desirability  for  the  final  system,  one  would  not  expect  such  a  rating  to  correlate  with 
either  measure  of  usability,  if  the  participants  are  treating  these  as  separate  dimensions  to  be  evaluated. 
On  the  other  hand,  one  would  expect  a  utility  judgement  on  whether  a  feature  is  an  improvement  over 
the  current  system  to  be  somewhat  influenced  by  its  usability. 

The  table  below  shows  the  Pearson  correlation  matrix  between  each  of  the  measures,  derived  from  the 
average  rating  across  all  participants  for  all  system  features.  Please  note  a  perfect  positive  correlation 
is  1.0  while  a  perfect  negative  correlation  is  -1.0.  No  correlation  is  0. 


Table  Dl:  Correlation  of  measures. 


Operate 

Effectively 

Desirable 

Feature 

Improvement 
over  now 

Easy  to  use  in 

trial 

Easy  to  use  in 
the  field 

Operate 

Effectively 

.82 

.59 

.73 

.73 

Desirable 

Feature 

.29 

.40 

.43 

Improvement 
over  now 

.78 

.74 

Easy  to  use  in 

trial 

.91 

These  data  provide  some  support  for  the  underlying  validity  of  the  measures.  Feature  desirability  is 
not  correlated  with  the  utility  measures,  nor  whether  the  particular  feature  is  seen  as  an  improvement. 
This  suggests  that  the  trial  participants  rate  this  aspect  of  utility  relatively  independently  of  other 
issues.  The  two  usability  measures  are  highly  correlated  as  one  might  expect,  and  less  well  correlated 
with  other  measures.  Also  expected  is  the  positive  relationship  between  whether  the  perception  of  the 
feature  as  an  improvement  over  current  capabilities  and  the  feature’s  usability.  Again,  because  of  the 
small  size  of  the  data  set,  these  findings  should  not  be  over- interpreted,  however,  they  do  give 
encouragement  to  believing  that  what  was  intended  to  be  measured  was  actually  measured. 

A  second  aspect  of  validity  is  external  validity,  which  encompasses  the  issue  of  generalisability,  that  is 
the  extent  to  which  the  results  of  an  evaluation  may  be  extended  to  other  groups  and  settings.  In  the 
case  of  the  TBCS  evaluation  performed  in  an  “office-like”  environment,  the  question  would  be  the 
extent  to  which  the  results  can  be  generalised  to  operational  C2  settings  and  different  personnel.  A 
principle  factor  that  will  influence  this  in  the  present  case  is  the  fidelity  of  the  test  environment  that  is, 
the  extent  to  which  it  captures  the  environment  influences  and  critical  tasks  for  which  TBCS  will  be 
applied  in  the  field.  We  have  attempted  to  address  the  issue  of  external  validity  in  two  ways:  first,  all 
of  the  system  features  are  evaluated  in  a  scenario-based  context  that  has  high  external  validity. 

Second,  we  have  specifically  required  trial  participants  to  distinguish  between  their  impression  of  the 
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system  in  the  context  of  the  trial  environment,  and  the  actual  field  conditions  for  its  intended  use. 
However,  it  is  unlikely  that  this  latter  approach  can  ensure  adequate  validity  for  extrapolating  the 
present  trial  data  to  actual  field  use  that  involves  working  in  moving  vehicles  under  full  exercise  or 
operational  conditions.  It  is  must  also  be  recognised  that  given  the  small  sample  size,  and  limited 
experience  of  some  of  the  trial  participants  in  the  role  they  were  assigned,  the  data  have  low  validity 
with  respect  to  the  different  operational  requirements  of  the  different  combat  team  positions 
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(U)  This  study  evaluated  the  utility  and  ease  of  use  of  features  of  the  Tactical  Battlefield 
Command  Systems  (TBCS)/Chameleon  using  participants  representing  command 
elements  of  a  combat  team.  Seven  participants  role-played  an  advance  to  contact 
scenario  developed  by  Joint  Command  Staff  Training  Centre  (JCSTC)  in  13  segments. 
Following  each  segment,  participants  provided  user  feedback  on  25  key  features  and  tools 
of  the  software. 

The  overall  results  indicated  that  the  features  and  tools  in  TBCS/Chameleon  are  seen  to 
be  generally  useful  by  the  combat  team  across  a  range  of  activities.  Many  specific 
features  currently  in  the  software,  as  well  as  future  features,  were  seen  to  have  particularly 
high  utility  and  have  the  potential  to  improve  situation  awareness,  reduce  workload, 
improve  communication  effectiveness  and  support  decision-making. 

However,  there  are  a  number  of  areas  in  which  the  utility  of  features  can  be  improved. 
Specific  recommendations  are  made  to  support  these  improvements  across  a  range  of 
features  including:  map  use,  communication  tools,  production  of  orders  and  access  to 
information. 

These  recommendations  concentrate  on  utility  issues  with  a  secondary  focus  on 
increasing  the  ease  of  use  of  some  features. 

The  user  review  process  should  continue  at  each  major  build  of  the  TBCS/Chameleon.  As 
the  development  moves  from  a  concept  based  development  to  a  fieldable  system  the  user 
reviews  should  move  from  utility  based  to  usability  based.  Tabletop  user  reviews  of 
concepts  will  also  assist  with  design  decisions  between  major  builds. 

(U)  Cette  etude  a  evalue  I’utilite  et  la  convivialite  des  fonctions  du  Systeme  tactique  de 
commandement  sur  le  champ  de  bataille  (STCCB)/Chameleon  en  faisant  appel  a  des 
participants  representant  les  elements  de  commandement  d’une  equipe  de  combat.  Sept 
participants  ont  simule  un  scenario  de  marche  a  I’ennemi  mise  au  point  par  le  Centre  de 
formation  de  commandement  et  d’etat  major  interarmees  (CFCEMI)  en  13  segments. 
Apres  chaque  segment,  les  participants  ont  fourni  une  retroaction  sur  25  caracteristiques 
et  outils  principaux  du  logiciel. 

Les  resultats  globaux  ont  indique  que  les  caracteristiques  et  outils  du  STCCB/Chameleon 
sont  consideres  comme  generalement  utiles  par  I’equipe  de  combat,  et  ce,  dans  differents 
champs  d’activite.  De  nombreuses  caracteristiques  actuelles  du  logiciel,  de  meme  que  des 
caracteristiques  envisagees,  sont  perques  comme  etant  particulierement  utiles  et  comme 
ayant  le  potentiel  d’ameliorer  la  connaissance  de  la  situation,  de  reduire  la  charge  de 
travail,  d’accroTtre  I’efficacite  des  communications  et  de  soutenir  la  prise  de  decisions. 
Cependant,  il  y  a  un  certain  nombre  des  caracteristiques  qui  pourraient  etre  ameliorees. 
Des  recommandations  specifiques  sont  formulees  en  vue  d’operer  ces  ameliorations  a 
diverses  caracteristiques  :  utilisation  de  cartes,  outils  de  communication,  production 
d’ordres  et  acces  a  I’information. 

Ces  recommandations  sont  axees  sur  des  questions  utilitaires  et  ont  pour  deuxieme 
centre  d’interet  la  convivialite  accrue  de  certaines  fonctions. 

Le  processus  d’examen  par  les  utilisateurs  devrait  se  poursuivre  avec  chacune  des 
nouvelles  versions  importantes  du  STCCB/Chameleon.  A  mesure  que  le  systeme  passe 
de  I’etape  conceptuelle  a  I’etape  de  I’utilisation  sur  le  terrain,  les  examens  par  les 
utilisateurs  devraient  etre  de  moins  en  moins  axes  sur  I’utilite  du  systeme  et  de  plus  en 
plus  sur  sa  convivialite.  L’examen  de  concepts  par  les  utilisateurs  au  moyen  de 
simulations  facilitera  egalement  la  prise  de  decisions  conceptuelles  entre  les  principales 


versions. 
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